When using functions exported by cdll.LoadLibrary, you're releasing the Global Interpreter Lock (GIL) as you enter the method. If you want to call python code, you need to re-acquire the lock.
e.g.
void someFunctionWithPython()
{
...
PyGILState_STATE state = PyGILState_Ensure();
printf("importing numpy...\n");
PyObject* numpy = PyImport_ImportModule("numpy");
if (numpy == NULL)
{
printf("Warning: error during import:\n");
PyErr_Print();
Py_Finalize();
PyGILState_Release(state);
exit(1);
}
PyObject* repr = PyObject_Repr(numpy);
PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
const char *bytes = PyBytes_AS_STRING(str);
printf("REPR: %s\n", bytes);
Py_XDECREF(repr);
Py_XDECREF(str);
PyGILState_Release(state);
return;
}
(python3.9-config --includes --ldflags --embed) -shared -o mylibwithpy.so mylibwithpy.c
$ LD_LIBRARY_PATH=. python driver.py
opening mylibwithpy.so...
.so object: <CDLL 'mylibwithpy.so', handle 1749f50 at 0x7fb603702fa0>
.so object's 'someFunctionWithPython': <_FuncPtr object at 0x7fb603679040>
calling someFunctionWithPython...
python alread initialized.
importing numpy...
REPR: <module 'numpy' from '/home/me/test/.venv/lib/python3.9/site-packages/numpy/__init__.py'>
Also if you look at PyDLL it says:
Instances of this class behave like CDLL instances, except that the Python GIL is not released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised.
So if you use PyDLL for your driver then you wouldn't need to re-acquire the lock in the C code:
from ctypes import PyDLL
if __name__ == "__main__":
print("opening mylibwithpy.so...");
my_so = PyDLL("mylibwithpy.so")
print(".so object: ", my_so)
print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython)
print("calling someFunctionWithPython...");
my_so.someFunctionWithPython()
UPDATE
Why do numpy's internal shared objects files not link to libpython3.8.so?
I believe numpy is setup this way because it expects to be called by the python interpreter where libpython will already be loaded and have the symbols made available.
That said, we can make the python libraries available for when mylibwithpy calls the import of numpy by using RTLD_GLOBAL.
The symbols defined by this shared object will be made available for symbol resolution of subsequently loaded shared objects.
The update to your code is simple:
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY | RTLD_GLOBAL);
Now all of the python libraries will be included because they are a dependency of mylibwithpy, meaning they will be available by the time that numpy loads its own shared libraries.
Alternatively, you could choose to load just libpythonX.Y.so with RTLD_GLOBAL to prior to loading mylibwithpy.so to minimize the amount symbols made globally available.
printf("opening libpython3.9.so...\n");
void* libpython3_so = dlopen("libpython3.9.so", RTLD_LAZY | RTLD_GLOBAL);
if (libpython3_so == NULL){
printf("an error occurred during loading libpython3.9.so: \n%s\n", dlerror());
exit(1);
}
printf("opening mylibwithpy.so...\n");
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY);
if (mylibwithpy_so == NULL){
printf("an error occurred during loading mylibwithpy.so: \n%s\n", dlerror());
exit(1);
}
Docker setup I used to recreate and test:
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
build-essential \
python3.9-dev \
python3.9-venv
RUN mkdir /workspace
WORKDIR /workspace
RUN python3.9 -m venv .venv
RUN .venv/bin/python -m pip install numpy
COPY . /workspace
RUN gcc -o mylibwithpy.so mylibwithpy.c -fPIC -shared \
$(python3.9-config --includes --ldflags --embed --cflags)
RUN gcc -o cdriver driver.c -L/usr/lib/x86_64-linux-gnu -Wall -ldl
ENV LD_LIBRARY_PATH=/workspace
# Then run: . .venv/bin/activate && ./cdriver
Answer from flakes on Stack OverflowHow would you properly use the Python/C API in a C shared library? - Stack Overflow
AskPython: Best Tutorials/Examples of the Python C API?
Is there a good guide for learning how to write native (c) code for Python?
Guide on how to use DeepSeek-v3 model with Cline
Videos
--excluding the official documentation. I'm looking for a good comprehensive tutorial that uses both examples and explains what's happening.
edit I have no idea why this is being downvoted. If you aren't interested, don't upvote it. This isn't spam, it's in the right category, and I'm just looking for advice. Just plain poor reddiquette.