embedding python in c++ , example - Stack Overflow
Embedding Python in C - Stack Overflow
How to Embed Python in C++?
Embedding Python in C Application - Stack Overflow
Hi,
I'm not sure if this is the place to post this, but here goes.
I'm currently trying to use the python interpreter as an embedded script engine inside another application written in C++. The overall goal is to use the python scripts to make the main application's behaviour dynamic, depending on which set of python files was loaded.
There are several requirements:
-
The c++ application exposes several of it's data structures so that they can be manipulated by python scripts. There are several discrete parts of the C++ code that should be exposed to the scripts as their own python module.
-
Multiple python scripts are loaded from the file system. Most of these define one or more functions that can then be called from the main application.
-
The application must be able to call the python functions from within multiple threads.
Most examples on the python website and boost's focus on either pure embedding or custom modules written in C/C++ that are then placed in the python interpreter's module path so I've not been able to find a comparable example. Consequently, I'm wondering if I'm using the right approach. I've run into afew problems related to modules and I'm not sure if what I'm doing is entirely the right way to go abou it. This is what I've done so far:
-
I've used Boost Python to expose several C++ classes and functions as python types. There are multiple modules with a few types and funtions each.
-
I've wrapped all interpreter related startup and shutdown code into a singleton class. On startup it calls Py_Initialize(), redirects sys.out and sys.err to the main application's logging, calls the init methods for each custom python module generated by boost python and calls PyEval_InitThreads to enable multithreading. The singleton keeps track of the main thread state and performs an orderly shutdown in it's destructor.
-
Once the interpreter is started, the application starts loading the python scripts from the file system. Each script file should be loaded as it's own module (I'm still thinking on how to do this) and uses only the embedded modules generated with Boost's python library.
-
Afterwards, whenever the main application needs to behave "dynamically", it calls a function within the python code. This is done by resolving the function by module name and function name from a handle to the main dictionary and then executing it (once again using boost python to find and call the function). Each call is wrapped within a packaged task and executed asynchronously on a thread pool so that the main application does not block on the global interpreter lock unless explicitly necessary. I've used PyGILState_Ensure() and PyGILState_Release() before each task to properly synchronize the calls.
Most of this works but I've ran into problems where import statements in python don't work while calling functions via the module name explicitely does work. For example: I'm creating a new object of type A in python. A is defined in module B, a module generated with boost python. If I explicitely construct B.A() it works without problems but whenever I try to import a name from module B I get a python error that it can't find the module. Does anyone know of a comprehensive example of loading custom modules into an embedded interpreter or, failing that, of a comprehensive guide on how to embed python into a C/C++ application (with or without boost) ?
Any pointers to good documentation or examples would be hugely appreciated.
Background: Hello! I am working on a project that will compute physics, graphics, and AI/ML tasks. I want to handle the physics and graphics calculations with C++ and the AI/ML calculations with Python. I have learned that you can embed Python in C++ recently but seem to be having trouble getting the embedding to work. I spent sometime away from computer science - about a year - and decided to comeback recently. I made the discovery that I like the theoretical side of computer science and want to conduct research. I am actually going to graduate school in computer science soon and I am very excited. I want to use this project to jump start my Ph.D.
Setup: Visual Studio Code on Windows 11
Questions: What is the best way to run and access Python scripts from C++ and how exactly can this be accomplished? I've tried inserting pybind11 as a submodule to my project git repository (I used this resource), but it has trouble finding certain header files, like Python.h - is there another solution that doesn't need pybind11? Is embedding Python in C++ even worth it? Should I just stick to using just one language or the other?
Thank you!