At the beginning, I wanted to say that there's no module required (at least no non-builtin one) for Py_InitializeEx, so the only requirement was python27.dll (BTW: python27.lib is not required, unless your colleagues want to link something against it - but that wouldn't be very easy w / o Python's Include dir).
I had this code (using Python 2.7.10 that I built using VStudio 10 (2010)):
#include <stdio.h>
#include <conio.h>
#include <Python.h>
int main()
{
int i = 0;
char *pyCode =
"s = \"abc\"\n"
"print s, 1234";
Py_InitializeEx(0);
i = PyRun_SimpleString(pyCode);
Py_Finalize();
printf("PyRun_SimpleString returned: %d\nPress a key to exit...\n", i);
_getch();
return 0;
}
It built fine, it ran OK from VStudio, and from the command line (after copying the .dll in its folder). But then I copied the .exe and .dll to another computer and when running, bang!!!
ImportError: No module named site
Considering that:
I have no PYTHON* env vars set in neither of the consoles on the 2 machines where I ran the .exe (with different results)
On both machines the Python installation is on the same path (I previously (years ago) modified it on the machine that doesn't work)
I don't know why it doesn't behave the same (one thing that I haven't check is that there might be some registry key on the machine that works?).
Note: site is a (.py(c)) module located under %PYTHON_INSTALL_DIR%\Lib.
Then, I browsed Python's source code and I came across [GitHub]: python/cpython - (v2.7.10) cpython/Python/pythonrun.c (function Py_InitializeEx - currently line #141) - this is how I'm going to refer a point in the source code):
if (!Py_NoSiteFlag)
initsite(); /* Module site */
while in initsite:
m = PyImport_ImportModule("site");
which is pretty obvious (Py_NoSiteFlag is 0).
Then I noticed that Py_NoSiteFlag is declared as extern __declspec(dllexport) ([MS.Learn]: Using extern to Specify Linkage, [MS.Learn]: dllexport, dllimport), so I modified my code to:
#include <stdio.h>
#include <conio.h>
#include <Python.h>
extern __declspec(dllimport) int Py_NoSiteFlag;
int main()
{
int i = 0;
char *pyCode =
"s = \"abc\"\n"
"print s, 1234";
Py_NoSiteFlag = 1;
Py_InitializeEx(0);
i = PyRun_SimpleString(pyCode);
Py_Finalize();
printf("PyRun_SimpleString returned: %d\nPress a key to exit...\n", i);
_getch();
return 0;
}
and it works! Yay!
So, at this point only the .dll is required in order to run a piece of code. But I imagine that your code is "a little bit" more complex than that (it has imports ([Python 2.Docs]: The import statement). To solve the import problem, you can use this nice module: [Python 2.Docs]: modulefinder - Find modules used by a script (part of Python 2.7's standard modules). To make use of it:
Save the code that you execute from C in a .py file
Run modulefinder against it
Here's an example for my code (pyCode contents in my C program, saved in a file).
code00.py:
s = "abc"
print(s, 1234)
Running:
${PYTHON_INSTALL_DIR}\python.exe -m modulefinder code00.py
yields:
Name File ---- ---- m __main__ code00.py
But, if I add an import os (which is a pretty common module) statement in the file, the above command yields:
Name File ---- ---- m StringIO %PYTHON_INSTALL_DIR%\lib\StringIO.py m UserDict %PYTHON_INSTALL_DIR%\lib\UserDict.py m __builtin__ m __future__ %PYTHON_INSTALL_DIR%\lib\__future__.py m __main__ a.py m _abcoll %PYTHON_INSTALL_DIR%\lib\_abcoll.py m _codecs m _collections m _functools m _hashlib %PYTHON_INSTALL_DIR%\DLLs\_hashlib.pyd m _heapq m _io m _locale m _random m _sre m _struct m _subprocess m _threading_local %PYTHON_INSTALL_DIR%\lib\_threading_local.py m _warnings m _weakref m _weakrefset %PYTHON_INSTALL_DIR%\lib\_weakrefset.py m abc %PYTHON_INSTALL_DIR%\lib\abc.py m array m atexit %PYTHON_INSTALL_DIR%\lib\atexit.py m bdb %PYTHON_INSTALL_DIR%\lib\bdb.py m binascii m cPickle m cStringIO m cmd %PYTHON_INSTALL_DIR%\lib\cmd.py m codecs %PYTHON_INSTALL_DIR%\lib\codecs.py m collections %PYTHON_INSTALL_DIR%\lib\collections.py m copy %PYTHON_INSTALL_DIR%\lib\copy.py m copy_reg %PYTHON_INSTALL_DIR%\lib\copy_reg.py m difflib %PYTHON_INSTALL_DIR%\lib\difflib.py m dis %PYTHON_INSTALL_DIR%\lib\dis.py m doctest %PYTHON_INSTALL_DIR%\lib\doctest.py m dummy_thread %PYTHON_INSTALL_DIR%\lib\dummy_thread.py P encodings %PYTHON_INSTALL_DIR%\lib\encodings\__init__.py m encodings.aliases %PYTHON_INSTALL_DIR%\lib\encodings\aliases.py m errno m exceptions m fnmatch %PYTHON_INSTALL_DIR%\lib\fnmatch.py m functools %PYTHON_INSTALL_DIR%\lib\functools.py m gc m genericpath %PYTHON_INSTALL_DIR%\lib\genericpath.py m getopt %PYTHON_INSTALL_DIR%\lib\getopt.py m gettext %PYTHON_INSTALL_DIR%\lib\gettext.py m hashlib %PYTHON_INSTALL_DIR%\lib\hashlib.py m heapq %PYTHON_INSTALL_DIR%\lib\heapq.py m imp m inspect %PYTHON_INSTALL_DIR%\lib\inspect.py m io %PYTHON_INSTALL_DIR%\lib\io.py m itertools m keyword %PYTHON_INSTALL_DIR%\lib\keyword.py m linecache %PYTHON_INSTALL_DIR%\lib\linecache.py m locale %PYTHON_INSTALL_DIR%\lib\locale.py P logging %PYTHON_INSTALL_DIR%\lib\logging\__init__.py m marshal m math m msvcrt m nt m ntpath %PYTHON_INSTALL_DIR%\lib\ntpath.py m opcode %PYTHON_INSTALL_DIR%\lib\opcode.py m operator m optparse %PYTHON_INSTALL_DIR%\lib\optparse.py m os %PYTHON_INSTALL_DIR%\lib\os.py m os2emxpath %PYTHON_INSTALL_DIR%\lib\os2emxpath.py m pdb %PYTHON_INSTALL_DIR%\lib\pdb.py m pickle %PYTHON_INSTALL_DIR%\lib\pickle.py m posixpath %PYTHON_INSTALL_DIR%\lib\posixpath.py m pprint %PYTHON_INSTALL_DIR%\lib\pprint.py m random %PYTHON_INSTALL_DIR%\lib\random.py m re %PYTHON_INSTALL_DIR%\lib\re.py m repr %PYTHON_INSTALL_DIR%\lib\repr.py m select %PYTHON_INSTALL_DIR%\DLLs\select.pyd m shlex %PYTHON_INSTALL_DIR%\lib\shlex.py m signal m sre_compile %PYTHON_INSTALL_DIR%\lib\sre_compile.py m sre_constants %PYTHON_INSTALL_DIR%\lib\sre_constants.py m sre_parse %PYTHON_INSTALL_DIR%\lib\sre_parse.py m stat %PYTHON_INSTALL_DIR%\lib\stat.py m string %PYTHON_INSTALL_DIR%\lib\string.py m strop m struct %PYTHON_INSTALL_DIR%\lib\struct.py m subprocess %PYTHON_INSTALL_DIR%\lib\subprocess.py m sys m tempfile %PYTHON_INSTALL_DIR%\lib\tempfile.py m textwrap %PYTHON_INSTALL_DIR%\lib\textwrap.py m thread m threading %PYTHON_INSTALL_DIR%\lib\threading.py m time m token %PYTHON_INSTALL_DIR%\lib\token.py m tokenize %PYTHON_INSTALL_DIR%\lib\tokenize.py m traceback %PYTHON_INSTALL_DIR%\lib\traceback.py m types %PYTHON_INSTALL_DIR%\lib\types.py P unittest %PYTHON_INSTALL_DIR%\lib\unittest\__init__.py m unittest.case %PYTHON_INSTALL_DIR%\lib\unittest\case.py m unittest.loader %PYTHON_INSTALL_DIR%\lib\unittest\loader.py m unittest.main %PYTHON_INSTALL_DIR%\lib\unittest\main.py m unittest.result %PYTHON_INSTALL_DIR%\lib\unittest\result.py m unittest.runner %PYTHON_INSTALL_DIR%\lib\unittest\runner.py m unittest.signals %PYTHON_INSTALL_DIR%\lib\unittest\signals.py m unittest.suite %PYTHON_INSTALL_DIR%\lib\unittest\suite.py m unittest.util %PYTHON_INSTALL_DIR%\lib\unittest\util.py m warnings %PYTHON_INSTALL_DIR%\lib\warnings.py m weakref %PYTHON_INSTALL_DIR%\lib\weakref.py Missing modules: ? _emx_link imported from os ? ce imported from os ? fcntl imported from subprocess, tempfile ? org.python.core imported from copy, pickle ? os.path imported from os, shlex ? os2 imported from os ? posix imported from os ? pwd imported from posixpath ? readline imported from cmd, pdb ? riscos imported from os ? riscosenviron imported from os ? riscospath imported from os
As you can see, there is an awfully lot of modules (I modified the output a little bit, instead of the actual path I placed the ${PYTHON_INSTALL_DIR} env var placeholder). In order for the Python code to work, you'll have to include all of those modules/packages in the installer.
Notes about modulefinder's output (that I've noticed while playing with it):
It searches for modules recursively, so here is the whole module dependency tree
It searches for import statements located in functions (so, not only the ones at module level)
It doesn't search for dynamic imports (e.g. [Python 2.Docs]: __import__(name[, globals[, locals[, fromlist[, level]]]]))
So, looking at the modules that are required by os, I'm not sure that taking out the site import from C, makes much of a difference.
IMPORTANT NOTE: To make sure your .exe works on any computer, you might consider including VStudio C Runtime Library or VCRTLib (msvcr##(#).dll: [MS.Learn]: Run-Time Library Reference) (where *#*s are placeholders for digits - representing VStudio version) in your installer.
Answer from CristiFati on Stack OverflowYour understanding is correct: invoking PyEval_InitThreads does, among other things, acquire the GIL. In a correctly written Python/C application, this is not an issue because the GIL will be unlocked in time, either automatically or manually.
If the main thread goes on to run Python code, there is nothing special to do, because Python interpreter will automatically relinquish the GIL after a number of instructions have been executed (allowing another thread to acquire it, which will relinquish it again, and so on). Additionally, whenever Python is about to invoke a blocking system call, e.g. to read from the network or write to a file, it will release the GIL around the call.
The original version of this answer pretty much ended here. But there is one more thing to take into account: the embedding scenario.
When embedding Python, the main thread often initializes Python and goes on to execute other, non-Python-related tasks. In that scenario there is nothing that will automatically release the GIL, so this must be done by the thread itself. That is in no way specific to the call that calls PyEval_InitThreads, it is expected of all Python/C code invoked with the GIL acquired.
For example, the main() might contain code like this:
Py_Initialize();
PyEval_InitThreads();
Py_BEGIN_ALLOW_THREADS
... call the non-Python part of the application here ...
Py_END_ALLOW_THREADS
Py_Finalize();
If your code creates threads manually, they need to acquire the GIL before doing anything Python-related, even as simple as Py_INCREF. To do so, use the following:
// Acquire the GIL
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
... call Python code here ...
// Release the GIL. No Python API allowed beyond this point.
PyGILState_Release(gstate);
There are two methods of multi threading while executing C/Python API.
1.Execution of different threads with same interpreter - We can execute a Python interpreter and share the same interpreter over the different threads.
The coding will be as follows.
main(){
//initialize Python
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print 'In Main, Today is',ctime(time())\n");
//to Initialize and acquire the global interpreter lock
PyEval_InitThreads();
//release the lock
PyThreadState *_save;
_save = PyEval_SaveThread();
// Create threads.
for (int i = 0; i<MAX_THREADS; i++)
{
hThreadArray[i] = CreateThread
//(...
MyThreadFunction, // thread function name
//...)
} // End of main thread creation loop.
// Wait until all threads have terminated.
//...
//Close all thread handles and free memory allocations.
//...
//end python here
//but need to check for GIL here too
PyEval_RestoreThread(_save);
Py_Finalize();
return 0;
}
//the thread function
DWORD WINAPI MyThreadFunction(LPVOID lpParam)
{
//non Pythonic activity
//...
//check for the state of Python GIL
PyGILState_STATE gilState;
gilState = PyGILState_Ensure();
//execute Python here
PyRun_SimpleString("from time import time,ctime\n"
"print 'In Thread Today is',ctime(time())\n");
//release the GIL
PyGILState_Release(gilState);
//other non Pythonic activity
//...
return 0;
}
- Another method is that, we can execute a Python interpreter in the main thread and, to each thread we can give its own sub interpreter. Thus every thread runs with its own separate , independent versions of all imported modules, including the fundamental modules - builtins, __main__ and sys.
The code is as follows
int main()
{
// Initialize the main interpreter
Py_Initialize();
// Initialize and acquire the global interpreter lock
PyEval_InitThreads();
// Release the lock
PyThreadState *_save;
_save = PyEval_SaveThread();
// create threads
for (int i = 0; i<MAX_THREADS; i++)
{
// Create the thread to begin execution on its own.
hThreadArray[i] = CreateThread
//(...
MyThreadFunction, // thread function name
//...); // returns the thread identifier
} // End of main thread creation loop.
// Wait until all threads have terminated.
WaitForMultipleObjects(MAX_THREADS, hThreadArray, TRUE, INFINITE);
// Close all thread handles and free memory allocations.
// ...
//end python here
//but need to check for GIL here too
//re capture the lock
PyEval_RestoreThread(_save);
//end python interpreter
Py_Finalize();
return 0;
}
//the thread functions
DWORD WINAPI MyThreadFunction(LPVOID lpParam)
{
// Non Pythonic activity
// ...
//create a new interpreter
PyEval_AcquireLock(); // acquire lock on the GIL
PyThreadState* pThreadState = Py_NewInterpreter();
assert(pThreadState != NULL); // check for failure
PyEval_ReleaseThread(pThreadState); // release the GIL
// switch in current interpreter
PyEval_AcquireThread(pThreadState);
//execute python code
PyRun_SimpleString("from time import time,ctime\n" "print\n"
"print 'Today is',ctime(time())\n");
// release current interpreter
PyEval_ReleaseThread(pThreadState);
//now to end the interpreter
PyEval_AcquireThread(pThreadState); // lock the GIL
Py_EndInterpreter(pThreadState);
PyEval_ReleaseLock(); // release the GIL
// Other non Pythonic activity
return 0;
}
It is necessary to note that the Global Interpreter Lock still persists and, in spite of giving individual interpreters to each thread, when it comes to python execution, we can still execute only one thread at a time. GIL is UNIQUE to PROCESS, so in spite of providing unique sub interpreter to each thread, we cannot have simultaneous execution of threads
Sources: Executing a Python interpreter in the main thread and, to each thread we can give its own sub interpreter
Multi threading tutorial (msdn)