Python searches the paths in sys.path in order (see http://docs.python.org/tutorial/modules.html#the-module-search-path). easy_install changes this list directly (see the last line in your easy-install.pth file):
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This basically takes whatever directories are added and inserts them at the beginning of the list.
Also see Eggs in path before PYTHONPATH environment variable.
Answer from cwa on Stack OverflowPython searches the paths in sys.path in order (see http://docs.python.org/tutorial/modules.html#the-module-search-path). easy_install changes this list directly (see the last line in your easy-install.pth file):
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This basically takes whatever directories are added and inserts them at the beginning of the list.
Also see Eggs in path before PYTHONPATH environment variable.
This page is a high Google result for "Python import order", so here's a hopefully clearer explanation:
- https://docs.python.org/library/sys.html#sys.path
- https://docs.python.org/tutorial/modules.html#the-module-search-path
As both of those pages explain, the import order is:
- Built-in python modules. You can see the list in the variable
sys.modules. - The
sys.pathentries. - The installation-dependent default locations.
And as the sys.path doc page explains, it is populated as follows:
- The first entry is the FULL PATH TO THE DIRECTORY of the file which
pythonwas started with (so/someplace/on/disk/> $ python /path/to/the/run.pymeans the first path is/path/to/the/, and likewise the path would be the same if you're in/path/to/> $ python the/run.py(it is still ALWAYS going to be set to the FULL PATH to the directory no matter if you gave python a relative or absolute file)), or it will be an empty string if python was started without a file aka interactive mode (an empty string means "current working directory for the python process"). In other words, Python assumes that the file you started wants to be able to do relative imports ofpackage/-foldersandblah.pymodules that exist within the same location as the file you started python with. - The other entries in
sys.pathare populated from thePYTHONPATHenvironment variable. Basically your global pip folders where your third-party python packages are installed (things likerequestsandnumpyandtensorflow).
So, basically: Yes, you can trust that Python will find your local package-folders and module files first, before any globally installed pip stuff.
Here's an example to explain further:
myproject/ # <-- This is not a package (no __init__.py file).
modules/ # <-- This is a package (has an __init__.py file).
__init__.py
foo.py
run.py
second.py
executed with: python /path/to/the/myproject/run.py
will cause sys.path[0] to be "/path/to/the/myproject/"
run.py contents:
import modules.foo as foo # will import "/path/to/the/myproject/" + "modules/foo.py"
import second # will import "/path/to/the/myproject/" + "second.py"
second.py contents:
import modules.foo as foo # will import "/path/to/the/myproject/" + "modules/foo.py"
EDIT:
You can run the following command to print a sorted list of all built-in module names. These are the things that load before ANY custom files/module folders in your projects. Basically these are names you must avoid in your own custom files:
python -c "import sys, json; print(json.dumps(sorted(list(sys.modules.keys())), indent=4))"
List as of Python 3.9.0:
"__main__",
"_abc",
"_bootlocale",
"_codecs",
"_collections",
"_collections_abc",
"_frozen_importlib",
"_frozen_importlib_external",
"_functools",
"_heapq",
"_imp",
"_io",
"_json",
"_locale",
"_operator",
"_signal",
"_sitebuiltins",
"_sre",
"_stat",
"_thread",
"_warnings",
"_weakref",
"abc",
"builtins",
"codecs",
"collections",
"copyreg",
"encodings",
"encodings.aliases",
"encodings.cp1252",
"encodings.latin_1",
"encodings.utf_8",
"enum",
"functools",
"genericpath",
"heapq",
"io",
"itertools",
"json",
"json.decoder",
"json.encoder",
"json.scanner",
"keyword",
"marshal",
"nt",
"ntpath",
"operator",
"os",
"os.path",
"pywin32_bootstrap",
"re",
"reprlib",
"site",
"sre_compile",
"sre_constants",
"sre_parse",
"stat",
"sys",
"time",
"types",
"winreg",
"zipimport"
So NEVER use any of those names for you .py files or your project module subfolders.
As you may know, sys.path is initialized from:
- the current directory
- your
PYTHONPATH - an installation-dependent default
However unfortunately that is only part of the story: setuptools creates easy-install.pth files, which also modify sys.path and worst of all they prepend packages and therefore totally mess up the order of directories.
In particular (at least on my system), there is /usr/local/lib/python2.7/dist-packages/easy-install.pth with the following contents:
import sys; sys.__plen = len(sys.path)
/usr/lib/python2.7/dist-packages
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This causes /usr/lib/python2.7/dist-packages to be prepended even before your PYTHONPATH!
What you could do is simply change the 2nd line in this file to
/usr/local/lib/python2.7/dist-packages
and you will get your desired priority.
However beware this file might be overwritten or changed again by a future setuptools invocation!
We encountered an almost identical situation and wanted to expand upon @kynan's response which is spot-on. In the case where you have such an easy-install.pth that you want to overcome, but which you cannot modify it (say you are a user with no root/admin access), you can do the following:
Set up an alternate python installation scheme
- e.g. we use a PYTHON HOME install (setting PYTHONUSERBASE)
Create a user/home site-packages
- You can do this by installing a package into the user env:
pip install <package> --user
- You can do this by installing a package into the user env:
Create a pth to set
sys.__egginsertto workaround the system/distribution easy-install.pth- Create a
$PYTHONUSERBASE/lib/python2.7/site-packages/fix_easy_install.pth - Containing:
import sys; sys.__egginsert = len(sys.path);
- Create a
This will set sys.__egginsert to point to the end of your sys.path including your usersite paths. When the nefarious system/dist easy-install.pth will then insert its items at the end of the system path.
I'm quite a beginner in Python and I found the answer of Anand was very good but quite complicated to me, so I try to reformulate :
1) insert and append methods are not specific to sys.path and as in other languages they add an item into a list or array and :
* append(item) add item to the end of the list,
* insert(n, item) inserts the item at the nth position in the list (0 at the beginning, 1 after the first element, etc ...).
2) As Anand said, python search the import files in each directory of the path in the order of the path, so :
* If you have no file name collisions, the order of the path has no impact,
* If you look after a function already defined in the path and you use append to add your path, you will not get your function but the predefined one.
But I think that it is better to use append and not insert to not overload the standard behaviour of Python, and use non-ambiguous names for your files and methods.
Because Python checks in the directories in sequential order, starting at the first directory in the sys.path list, until it finds the .py file it was looking for.
Normally, the current directory or the directory of the script is the first element in the list, unless you modify it (like you did). From documentation:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
So, most probably, you had a .py file with the same name as the module you were trying to import from, in the current directory (where the script was being run from).
Also, something to note about ImportErrors - Let's say the import error says:
ImportError: No module named main. This doesn't mean main.py is overwritten - if it was overwritten it would still exist and we would not have issues trying to read it. It's some module above this that got overwritten with a .py or some other file.
For example, consider a directory structure that looks like this:
- test
- shared
- __init__.py
- phtest.py
- testmain.py
Now from testmain.py, I call from shared import phtest, which works fine.
Now let's say I introduce a shared.py in test directory:
- test
- shared
- __init__.py
- phtest.py
- testmain.py
- shared.py
Now when I try to do from shared import phtest from testmain.py, I get the error:
ImportError: cannot import name 'phtest'
As you can see above, the file that introduced the issue is shared.py, not phtest.py.
A directory without an __init__.py (flask/) file is a namespace package, not a regular package (flask/__init__.py) or module (flask.py). While namespace packages are recorded in the order of sys.path, they are only used if neither a regular package or module is found at all.
PEP 420 -- Implicit Namespace Packages
...
Specification
...
While looking for a module or package named "foo", for each directory in the parent path:
- If
<directory>/foo/__init__.pyis found, a regular package is imported and returned.- If not, but
<directory>/foo.{py,pyc,so,pyd}is found, a module is imported and returned. The exact list of extension varies by platform and whether the-> Oflag is specified. The list here is representative.- If not, but
<directory>/foois found and is a directory, it is recorded and the scan continues with the next directory in the parent path.- Otherwise the scan continues with the next directory in the parent path.
If the scan completes without returning a module or package, and at least one directory was recorded, then a namespace package is created.
...
See the sys.path documentation regarding imports. Flask is now part of python 3 installations.
When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations.