Python searches the paths in sys.path in order (see http://docs.python.org/tutorial/modules.html#the-module-search-path). easy_install changes this list directly (see the last line in your easy-install.pth file):
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This basically takes whatever directories are added and inserts them at the beginning of the list.
Also see Eggs in path before PYTHONPATH environment variable.
Answer from cwa on Stack OverflowPython searches the paths in sys.path in order (see http://docs.python.org/tutorial/modules.html#the-module-search-path). easy_install changes this list directly (see the last line in your easy-install.pth file):
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This basically takes whatever directories are added and inserts them at the beginning of the list.
Also see Eggs in path before PYTHONPATH environment variable.
This page is a high Google result for "Python import order", so here's a hopefully clearer explanation:
- https://docs.python.org/library/sys.html#sys.path
- https://docs.python.org/tutorial/modules.html#the-module-search-path
As both of those pages explain, the import order is:
- Built-in python modules. You can see the list in the variable
sys.modules. - The
sys.pathentries. - The installation-dependent default locations.
And as the sys.path doc page explains, it is populated as follows:
- The first entry is the FULL PATH TO THE DIRECTORY of the file which
pythonwas started with (so/someplace/on/disk/> $ python /path/to/the/run.pymeans the first path is/path/to/the/, and likewise the path would be the same if you're in/path/to/> $ python the/run.py(it is still ALWAYS going to be set to the FULL PATH to the directory no matter if you gave python a relative or absolute file)), or it will be an empty string if python was started without a file aka interactive mode (an empty string means "current working directory for the python process"). In other words, Python assumes that the file you started wants to be able to do relative imports ofpackage/-foldersandblah.pymodules that exist within the same location as the file you started python with. - The other entries in
sys.pathare populated from thePYTHONPATHenvironment variable. Basically your global pip folders where your third-party python packages are installed (things likerequestsandnumpyandtensorflow).
So, basically: Yes, you can trust that Python will find your local package-folders and module files first, before any globally installed pip stuff.
Here's an example to explain further:
myproject/ # <-- This is not a package (no __init__.py file).
modules/ # <-- This is a package (has an __init__.py file).
__init__.py
foo.py
run.py
second.py
executed with: python /path/to/the/myproject/run.py
will cause sys.path[0] to be "/path/to/the/myproject/"
run.py contents:
import modules.foo as foo # will import "/path/to/the/myproject/" + "modules/foo.py"
import second # will import "/path/to/the/myproject/" + "second.py"
second.py contents:
import modules.foo as foo # will import "/path/to/the/myproject/" + "modules/foo.py"
EDIT:
You can run the following command to print a sorted list of all built-in module names. These are the things that load before ANY custom files/module folders in your projects. Basically these are names you must avoid in your own custom files:
python -c "import sys, json; print(json.dumps(sorted(list(sys.modules.keys())), indent=4))"
List as of Python 3.9.0:
"__main__",
"_abc",
"_bootlocale",
"_codecs",
"_collections",
"_collections_abc",
"_frozen_importlib",
"_frozen_importlib_external",
"_functools",
"_heapq",
"_imp",
"_io",
"_json",
"_locale",
"_operator",
"_signal",
"_sitebuiltins",
"_sre",
"_stat",
"_thread",
"_warnings",
"_weakref",
"abc",
"builtins",
"codecs",
"collections",
"copyreg",
"encodings",
"encodings.aliases",
"encodings.cp1252",
"encodings.latin_1",
"encodings.utf_8",
"enum",
"functools",
"genericpath",
"heapq",
"io",
"itertools",
"json",
"json.decoder",
"json.encoder",
"json.scanner",
"keyword",
"marshal",
"nt",
"ntpath",
"operator",
"os",
"os.path",
"pywin32_bootstrap",
"re",
"reprlib",
"site",
"sre_compile",
"sre_constants",
"sre_parse",
"stat",
"sys",
"time",
"types",
"winreg",
"zipimport"
So NEVER use any of those names for you .py files or your project module subfolders.
Hello, I built and packaged python from source code for my distro (cause the maintainers still don't have the latest version). The built and install was succeed but I need to set PYTHONPATH in order to make some modules get detected. The PYTHONPATH is "/usr/lib/python3.11/:/usr/lib64/python3.11/:/usr/lib/python3.11/site-packages/" . How the PYTHONPATH isn't configured / setted automatically ? And is that normal ? thanks.
Difference in sys.path order, depending on how run
python - Is the order reported by 'sys.path' the order in which packages are searched? - Stack Overflow
python - PYTHONPATH order on Ubuntu 14.04 - Stack Overflow
python - change the priority of pythonpath - Stack Overflow
As we cannot explore into your system, I am trying to analysis your first question by illustrating how sys.path is initialized. Available references are where-does-sys-path-starts and pyco-reverse-engineering(python2.6).
The sys.path comes from the following variables(in order):
$PYTHONPATH(highest priority)sys.prefix-ed stdlibsys.exec_prefix-ed stdlibsite-packages*.pthin site-packages (lowest priority)
Now let's describe each of these variables:
$PYTHONPATH, this is just a system environment variable.- & 3.
sys.prefixandsys.exec_prefixare determined before any python script is executed. It is actually coded in the source Module/getpath.c.
The logic is like this:
IF $PYTHONHOME IS set:
RETURN sys.prefix AND sys.exec_prefix as $PYTHONHOME
ELSE:
current_dir = directory of python executable;
DO:
current_dir = parent(current_dir)
IF FILE 'lib/pythonX.Y/os.py' EXSITS:
sys.prefix = current_dir
IF FILE 'lib/pythonX.Y/lib-dynload' EXSITS:
sys.exec_prefix = current_dir
IF current_dir IS '/':
BREAK
WHILE(TRUE)
IF sys.prefix IS NOT SET:
sys.prefix = BUILD_PREFIX
IF sys.exec_prefix IS NOT SET:
sys.exec_prefix = BUILD_PREFIX
& 5.
site-packagesand*.pthare added by import ofsite.py. In this module you will find the docs:This will append site-specific paths to the module search path. On Unix (including Mac OSX), it starts with sys.prefix and sys.exec_prefix (if different) and appends lib/python/site-packages as well as lib/site-python. ... ...
For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python/dist-packages, Debian addons install into /usr/{lib,share}/python/dist-packages. /usr/lib/python/site-packages is not used.
A path configuration file is a file whose name has the form .pth; its contents are additional directories (one per line) to be added to sys.path. ... ...
And a code snippet for important function getsitepackages:
sitepackages.append(os.path.join(prefix, "local/lib",
"python" + sys.version[:3],
"dist-packages"))
sitepackages.append(os.path.join(prefix, "lib",
"python" + sys.version[:3],
"dist-packages"))
Now I try to fig out where may be this odd problem comes from:
$PYTHONPATH, impossible, because it is empty both A and Bsys.prefixandsys.exec_prefix, maybe, please check them and as well as$PYTHONHOMEsite.py, maybe, check the file.
The sys.path output of B is quite odd, dist-package (site-package) goes before sys.exec_prefix (lib-dynload). Please try to investigate each step of sys.path initialization of machine B, you may find out something.
Very sorry that I cannot replicate your problem. By the way, about your question title, I think SYS.PATH is better than PYTHONPATH, which makes me misinterpretation as $PYTHONPATH at first glance.
If you peek into Python's site.py, which you can by opening /usr/lib/python2.7/site.py in a text editor.
The sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python/dist-packages, the global addons install into /usr/{lib,share}/python/dist-packages.
You can change the order by overriding this:
def getsitepackages():
"""Returns a list containing all global site-packages directories
(and possibly site-python).
For each directory present in the global ``PREFIXES``, this function
will find its `site-packages` subdirectory depending on the system
environment, and will return a list of full paths.
"""
sitepackages = []
seen = set()
for prefix in PREFIXES:
if not prefix or prefix in seen:
continue
seen.add(prefix)
if sys.platform in ('os2emx', 'riscos'):
sitepackages.append(os.path.join(prefix, "Lib", "site-packages"))
elif os.sep == '/':
sitepackages.append(os.path.join(prefix, "local/lib",
"python" + sys.version[:3],
"dist-packages"))
sitepackages.append(os.path.join(prefix, "lib",
"python" + sys.version[:3],
"dist-packages"))
else:
sitepackages.append(prefix)
sitepackages.append(os.path.join(prefix, "lib", "site-packages"))
if sys.platform == "darwin":
# for framework builds *only* we add the standard Apple
# locations.
from sysconfig import get_config_var
framework = get_config_var("PYTHONFRAMEWORK")
if framework:
sitepackages.append(
os.path.join("/Library", framework,
sys.version[:3], "site-packages"))
return sitepackages
Yes, you can import sys and manipulate sys.path as an ordinary list at runtime. You can rearrange what's there, or just insert at the beginning (sys.path.insert(0, 'path')). Do this before your import statement. If this will cause problems elsewhere, put it back after your import statement.
Note, this is fairly hacky. But it sounds like you might have a case for it, although I have not looked at these specific tools together.
Edit: this is more relevant if you want to control the python path at the level of individual imports within the course of one execution of python. If you want to control the path at the level of one full execution of Python, you can also set the python path on the command line for just that execution like this:
PYTHONPATH=/replacement/path/here python your_script.py
This is more verbose (unless you wrap it in a shell script or alias) than just calling python, but it lets you control the path one script at a time, wheres putting it into .bashrc/.bash_profile or similar changes it for your whole shell session.
Ordering the entries in PYTHONPATH is partly right, but the system paths don't seem to get included until you run python, and then they get put at the end. So to put a system path in front, explicitly add it:
export PYTHONPATH="[path/to/system/files]:$PYTHONPATH"
It is kind of a hack, because the system path you add will be duplicated in sys.path. But it works.
As you may know, sys.path is initialized from:
- the current directory
- your
PYTHONPATH - an installation-dependent default
However unfortunately that is only part of the story: setuptools creates easy-install.pth files, which also modify sys.path and worst of all they prepend packages and therefore totally mess up the order of directories.
In particular (at least on my system), there is /usr/local/lib/python2.7/dist-packages/easy-install.pth with the following contents:
import sys; sys.__plen = len(sys.path)
/usr/lib/python2.7/dist-packages
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This causes /usr/lib/python2.7/dist-packages to be prepended even before your PYTHONPATH!
What you could do is simply change the 2nd line in this file to
/usr/local/lib/python2.7/dist-packages
and you will get your desired priority.
However beware this file might be overwritten or changed again by a future setuptools invocation!
We encountered an almost identical situation and wanted to expand upon @kynan's response which is spot-on. In the case where you have such an easy-install.pth that you want to overcome, but which you cannot modify it (say you are a user with no root/admin access), you can do the following:
Set up an alternate python installation scheme
- e.g. we use a PYTHON HOME install (setting PYTHONUSERBASE)
Create a user/home site-packages
- You can do this by installing a package into the user env:
pip install <package> --user
- You can do this by installing a package into the user env:
Create a pth to set
sys.__egginsertto workaround the system/distribution easy-install.pth- Create a
$PYTHONUSERBASE/lib/python2.7/site-packages/fix_easy_install.pth - Containing:
import sys; sys.__egginsert = len(sys.path);
- Create a
This will set sys.__egginsert to point to the end of your sys.path including your usersite paths. When the nefarious system/dist easy-install.pth will then insert its items at the end of the system path.