Why not use Jython? The only downside I can immediately think of is if your library uses CPython native extensions.
EDIT: If you can use Jython now but think you may have problems with a later version of the library, I suggest you try to isolate the library from your app (e.g. some sort of adapter interface). Go with the simplest thing that works for the moment, then consider JNI/CPython/etc if and when you ever need to. There's little to be gained by going the (painful) JNI route unless you really have to.
Answer from Jon Skeet on Stack OverflowWhy not use Jython? The only downside I can immediately think of is if your library uses CPython native extensions.
EDIT: If you can use Jython now but think you may have problems with a later version of the library, I suggest you try to isolate the library from your app (e.g. some sort of adapter interface). Go with the simplest thing that works for the moment, then consider JNI/CPython/etc if and when you ever need to. There's little to be gained by going the (painful) JNI route unless you really have to.
Frankly most ways to somehow run Python directly from within JVM don't work. They are either not-quite-compatible (new release of your third party library can use python 2.6 features and will not work with Jython 2.5) or hacky (it will break with cryptic JVM stacktrace not really leading to solution).
My preferred way to integrate the two would use RPC. XML RPC is not a bad choice here, if you have moderate amounts of data. It is pretty well supported — Python has it in its standard library. Java libraries are also easy to find. Now depending on your setup either Java or Python part would be a server accepting connection from other language.
A less popular but worth considering alternative way to do RPCs is Google protobuffers, which have 2/3 of support for nice rpc. You just need to provide your transport layer. Not that much work and the convenience of writing is reasonable.
Another option is to write a C wrapper around that pieces of Python functionality that you need to expose to Java and use it via JVM native plugins. You can ease the pain by going with SWIG SWIG.
Essentially in your case it works like that:
- Create a SWIG interface for all method calls from Java to C++.
- Create C/C++ code that will receive your calls and internally call python interpreter with right params.
- Convert response you get from python and send it via swig back to your Java code.
This solution is fairly complex, a bit of an overkill in most cases. Still it is worth doing if you (for some reason) cannot afford RPCs. RPC still would be my preferred choice, though.
Why not use Jython? The only downside I can immediately think of is if your library uses CPython native extensions.
EDIT: If you can use Jython now but think you may have problems with a later version of the library, I suggest you try to isolate the library from your app (e.g. some sort of adapter interface). Go with the simplest thing that works for the moment, then consider JNI/CPython/etc if and when you ever need to. There's little to be gained by going the (painful) JNI route unless you really have to.
Answer from Jon Skeet on Stack OverflowYou could try this way:
- Use Jython instead of CPython to write Python code http://www.jython.org.
- Integrate Jython code with Java code through Apache Bean Scripting Framework http://commons.apache.org/bsf/
If you definitely need to use CPython, then Apache Trift could be interesting for you: http://thrift.apache.org/ So you could make additional scalable abstraction layer and integrate your Java code with different languages (not only Python)
If you need a really low-level interface you could look at JNI http://java.sun.com/docs/books/jni/ for investigation. But I think it will take a lot of time to integrate your code with CPython using JNI.
I've used JPype in a similar instance with decent results. The main task would be to write wrappers to translate your java api into a more pythonic api, since raw JPype usage is hardly any prettier than just writing java code.
My project uses some packages that are available only in Python and heavily rely on C libraries. The project also greatly benefits from Java libraries and the JVM. What's the optimal way to call Python functions from Java?
I tried:
-
Small web-services: overhead to serialize data, start and stop the services. Also debugging is harder and implementing each new function is now double the effort.
-
Jpy: a library that runs an interpreter in the JVM. Spare the service start/stop, but: isn't really feasible for more than a single-liner, data translation between Java and Python is cumbersome, and I also encountered runtime segmentation fault errors.
Any other options?
The project is in the machine-learning domain, so involves exchanging large numeric arrays and text. In some cases the execution switches back and forth between the platforms.
Interpreters Written in C89 with Reflection, Who Knew?
I have a feeling you are looking for an explanation of the mechanism and not a link to the API or instructions on how to code it. So, as I understand it . . .
The main interpreter is typically written in C and is dynamically linked. In a dynamically linked environment, even C89 has a certain amount of reflective behavior. In particular, the dlopen(3) and dlsym(3) calls will load a dynamic (typically ELF) library and look up the address of a symbol named by a string. Give that address, the interpreter can call a function. Even if statically linked, the interpreter can know the address of C functions whose names are compiled into it.
So then, it's just a simple matter of having the interpreted code tell the interpreter to call a particular native function in a particular native library.
The mechanism can be modular. An extension library for the interpreter, written in the script, can itself invoke the bare hooks for dlopen(3) and dlsym(3) and hook up to a new library that the interpreter never knew about.
For passing simple objects by value, a few prototype functions will typically allow various calls. But for structured data objects (imagine stat(2)) the wrapper module needs to know the layout of the data. At some point, either when packaging the extension module or when installing it, a C interface module includes the appropriate header files and in conjunction with handwritten code constructs an interface object. This is why you may need to install something like libsqlite3-dev even if you already had sqlite3 on your system; only the -dev package has the .h files needed to recompile the linkage code.
I suppose we could sum this up by saying: "it's done with brute force and ignorance". :-)
The main general concept is known as FFI, "Foreign Function Interface" -- for Java it's JNI, for Python it's the "Python C API", for Perl it's XS, etc, etc, but I think it's important to give you the general term of art to help you research it more thoroughly.
Given a FFI, you can write (e.g.) C programs that respect it directly, and/or you can have code generators that produce such C code from metainformation they receive and/or introspect from code written in other languages (often with some help, e.g., to drive the SWIG code generator you typically decorate the info that's in a .h C header file with extra info that's SWIG-specific to get a better wrapper).
There are also special languages such as Cython, an "extended subset" of Python that's geared towards easy generation of FFI code while matching much of Python's syntax and semantics -- may often be the easiest way for mostly-Python programmers to write a Python extension module that compiles down to speedy machine code and maybe uses some existing C-callable libraries.
The ctypes approach is different from the traditional FFI approaches, though it self-describes as a "foreign function library for Python" -- it relies on the foreign code being available in a DLL (or equivalent, such as an .so dynamic library in Linux), and generates and executes code at run-time to reach into such dynamically loaded C code (typically all done via explicit programming in Python -- I don't know of ctypes wrappers based on introspection and ctypes-code generation, yet). Handy to avoid having to install anything special for simple tasks of accessing existing DLLs with Python, but I think it doesn't scale up as well as the FFI "linker-based" approaches (as it requires more runtime exertion, etc, etc). I don't know of any other implementation of such an approach, targeting other languages, beyond ctypes for Python (I imagine some do exist, given today's prevalence of DLL and .so packaging, and would be curious to learn about them).