I would avoid Spring-related frameworks at all cost. Look into Apache Airflow, AirByte and Apache Luigi for ETL/ELT frameworks that aren't a mess.

You can call the python progam as an executable.

A toy python program to do so is:

import java.io.*;
class Interact {

  static public void main(String[] args) throws IOException {
    Process p = Runtime.getRuntime().exec("python3 python_program.py arg1 arg2");
     BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream())); 
    String line;
    while ((line = in.readLine()) != null) {
        System.out.println(line);
    }
  }
}

if python_program.py is the following

import sys
if __name__ == "__main__"
     print("my args are",sys.argv[1:])

the result should be

my args are ['arg1', 'arg2']

of course on a more complete program you should read the stderr, check the return code etc...

This feels very vibe coded to me. Single commit and why is pycache committed? 😂 in draft.md Certainly! Let's further refine the design to provide a detailed view of the individual steps within the Processing Workflow Configuration Component, including specific data processing steps for each microservice.

I think Python's primary advantages of Java for ETL include: Same language used for ETL/data ingestion/data wrangling/etc is then used for data analysis. This means that the data consumer (data scientist, etc) can easily read the code to see what's going on with a transform, they can easily help write it, and their analytical pipelines can be delivered to the ETL team to support. Related to the previous point - Python is a more popular language with data scientists, who may be calling the shots about some ETL. Python is a concise and high-level language. Which is exactly what you want for most field transformations. It's easier to repurpose developers with other backgrounds into competent python developers than java developers. But Python also has some disadvantages to Java for ETL: It's slower, so you need to leverage pypy, parallelism, etc if you have large volumes. And you still might not be as fast as java, depending. Large, older enterprises are more likely to be java shops, and have a lot of java knowledge & bodies laying around. Some componentry is written in jvm languages (spark, hadoop), which may encourage some teams to stick with java. Static typing is all the rage these days. Some shops are really into it. Python is just getting aboard with gradual typing - which most people aren't yet aware of.

ActFramework is a full stack Java MVC framework. The experience of using Act is very much like using a combination of SpringBoot and Play!Framework 1.x. It got the best parts of both products without the cons.ActFramework is a full stack Java MVC framework. The experience of using Act is very much like using a combination of SpringBoot and Play!Framework 1.x. It got the best parts of both products without the cons.Performance: There are two aspects of performance: how quick a developer can deliver a feature and how fast the app is running on the product server. Act is designed to deliver excellent results in both aspects. With unbeatable developing experience, Act makes it very easy to release a feature; on the other side Act is very fast in runtime. Check out this 3rd party benchmark result. | Hot reload: You never restart while you're working on your Act application. Act's hot reload feature is fast and stable, it makes you feel like dealing with scripting language frameworks like Django or NodeJs. Watch this video and feel it. | Scalability: Act is built as a stateless framework. It supports horizontal scale. | Secure: Act is built as a secure product. It provides built-in CSRF/XSS prevention mechanism. And act-aaa makes it very easy to implement Authentication/Authorization/Auditing in your app. | Superb RESTful support: Act makes creating RESTful service a kids game. It features AdaptiveRecord (allows front end to drive the data structure), JSON response control (just declare the fields you want to present or hide) and RESTful URL routing with path variables. | Easy to deploy: ActFramework is not a servlet framework and there are no requirements on containers/app servers. It has a small package size (a helloworld distribution package size is less than 20 MB), a small memory feet print (a helloworld app heap usage is less than 20MB) and a fast boot up speed (a helloworld app starts in less than 3s). | Templating: Act's view architecture is very flexible and support using multiple view engines in your app. The default template engine is Rythm, a very developer friendly and powerful template engine. Act also support other templating solutions including freemarker, velocity, thymeleaf, and mustache via plugins. | Configuration for multiple environments: Act supports load configuration from a common dir and then overwriting it from a profile dir. Makes it very easy to manage configurations in different environments (e.g., dev, uat, sit, prod) Watch this video to see the innovative way Act delivers its configuration support. | Concise and expressive: Act does not require you to put Annotation when it is able to infer the intention from other parts of the code, i.e., you don't use `@PathVariable` or `@RequestParam` to tell Act the binding parameter name. And you don't need a `ModelMap` to bind variables to render argument names. Act has sophisticated byte class scanner to detect the variable names to do bindings automatically. | Comply to standards: Act's IoC is built on top of Genie, a fast dependency injection library that fully supports JSR330, and Act's validation solution is built on top of JSR303. Act is NOT an odd framework to most Java developers. Unlike Play1.x, ACT applications follow the standard maven project structure and it is very easy to integrate other Java libraries. | Database access: Act's DB layer is extremely easy to use. It supports SQL databases (through ebean orm) and MongoDB (through morphia). Using multiple datasource can never be that easy with Act's DB layer. Go here for more information on this. | Fast and flexible routing: You can configure your routing in either Spring MVC/Jersey style with annotation or Play style with route table or a combination of both. Act's routing supports RESTful URL path variables, optionally validated with regular expressions. | Incomplete microservice support: Although Act is built to be a great framework that supports microservice development, it lacks some of the key features at the moment, like sending requests to other microservices from within the app, service governance, and messaging handling. | Very small community: As of February 2017, Act is a brand new framework (even though the project started at the end of 2014). Community is still forming. | Functional testing support still under development: Developers are still working on innovative functional testing support for Act. | Documentation is still being written: Act's documentation is still under development.