I'm quite a beginner in Python and I found the answer of Anand was very good but quite complicated to me, so I try to reformulate :
1) insert and append methods are not specific to sys.path and as in other languages they add an item into a list or array and :
* append(item) add item to the end of the list,
* insert(n, item) inserts the item at the nth position in the list (0 at the beginning, 1 after the first element, etc ...).
2) As Anand said, python search the import files in each directory of the path in the order of the path, so :
* If you have no file name collisions, the order of the path has no impact,
* If you look after a function already defined in the path and you use append to add your path, you will not get your function but the predefined one.
But I think that it is better to use append and not insert to not overload the standard behaviour of Python, and use non-ambiguous names for your files and methods.
I'm quite a beginner in Python and I found the answer of Anand was very good but quite complicated to me, so I try to reformulate :
1) insert and append methods are not specific to sys.path and as in other languages they add an item into a list or array and :
* append(item) add item to the end of the list,
* insert(n, item) inserts the item at the nth position in the list (0 at the beginning, 1 after the first element, etc ...).
2) As Anand said, python search the import files in each directory of the path in the order of the path, so :
* If you have no file name collisions, the order of the path has no impact,
* If you look after a function already defined in the path and you use append to add your path, you will not get your function but the predefined one.
But I think that it is better to use append and not insert to not overload the standard behaviour of Python, and use non-ambiguous names for your files and methods.
Because Python checks in the directories in sequential order, starting at the first directory in the sys.path list, until it finds the .py file it was looking for.
Normally, the current directory or the directory of the script is the first element in the list, unless you modify it (like you did). From documentation:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
So, most probably, you had a .py file with the same name as the module you were trying to import from, in the current directory (where the script was being run from).
Also, something to note about ImportErrors - Let's say the import error says:
ImportError: No module named main. This doesn't mean main.py is overwritten - if it was overwritten it would still exist and we would not have issues trying to read it. It's some module above this that got overwritten with a .py or some other file.
For example, consider a directory structure that looks like this:
- test
- shared
- __init__.py
- phtest.py
- testmain.py
Now from testmain.py, I call from shared import phtest, which works fine.
Now let's say I introduce a shared.py in test directory:
- test
- shared
- __init__.py
- phtest.py
- testmain.py
- shared.py
Now when I try to do from shared import phtest from testmain.py, I get the error:
ImportError: cannot import name 'phtest'
As you can see above, the file that introduced the issue is shared.py, not phtest.py.
Why does sys.path.insert change the working directory and how to make it stop?
python - Why use sys.path.append(path) instead of sys.path.insert(1, path)? - Stack Overflow
Python files included by `sys.path.insert()` are missing on the final executable
First argument of sys.path.insert in python - Stack Overflow
I do sys.path.insert(0, '/other/dir') in order to import modules from directories which are not in subdirectories of the current project/far away but it is changing the working directory to that directory.
I never noticed that behavior tile recently.
I prefer to do insert rather than installing modules for system wide imports because I tried that once and since I often am changing the code in those modules I found it a nuisance to constantly be having to re-install the module every time I made a change. And that is for only one. It would really get annoying if there were several files which would need reinstalling every time the slightest change was made.
I am currently having to change the working directory back to the origininal one again with os but why does it do it in the first place and how to prevent?
I am often importing modules from far away and unrelated(to the original script) places so want to know how to do this with least fuss- the advised make them installable suggestion was way more trouble though due to aforementioned reason.
How python finds its modules
Strictly taken, a module is a single python file, while a package is a folder containing python files, accompanied by a (can be empty) file named __init__.py, to tell python it is a package to import modules from. In both cases, modules need their .py extension, but importing them is done without (see further below).
By default, Python looks for its modules and packages in $PYTHONPATH.
To find out what is included in $PYTHONPATH, run the following code in python (3):
import sys
print(sys.path)
How to add a directory
Occasionally
From within a python file, you can add path(s) occasionally to the default path by adding the following lines in the head section of your python application or script:
import sys
sys.path.insert(0, "/path/to/your/package_or_module")
For example:
if I have a folder: /home/myname/pythonfiles, and I want to import the file module_1.py, located in that directory, I add this to the head section of my code:
import sys
sys.path.insert(0, "/home/myname/pythonfiles")
And I can simply import the file module_1.py by:
import module_1
When I create a package and want to import module(s) from the package, I need to create a folder in $PYTHONPATH, containing the modules, accompanied by a (can be empty) file called __init__.py
For example:
To import from a package (folder) called my_package in /home/myname/pythonfiles , add the /home/myname/pythonfiles path to your $PYTHONPATH, like in example 1, and import the module called module_2.py (inside the package folder) simply with: `
from <packagename> import module_2
Adding directories to $PYTHONPATH permanently:
Add the following line to your ~/.profile file.
export PYTHONPATH=$PYTHONPATH:/path/you/want/to/add
Subdirectories
From within a package, subdirectories are not included just like that; you need to "chain" the directories. To import a module module_3.py, inside folder subfolder inside folder packagename:
import packagename.subfolder.module_3
Given the fact that all subfolders in the package include their own __init__.py file.
When a module is in the same directory as the script or application
There is no need to insert the path to a module when it is in the same directory as the script or application, it is automatically added.
Example:
If I have a folder, containing script.py and module.py, I can simply import the module by:
import module
The correct way to use relative import is:
from ..Common import foo1
And you also need a __init__.py in all your folders.
If you really need to use sys.path.insert, consider leaving sys.path[0] as it is:
sys.path.insert(1, path_to_dev_pyworkbooks)
This could be important since 3rd party code may rely on sys.path documentation conformance:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.
If you have multiple versions of a package / module, you need to be using virtualenv (emphasis mine):
virtualenvis a tool to create isolated Python environments.The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into
/usr/lib/python2.7/site-packages(or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global
site-packagesdirectory? For instance, on a shared host.In all these cases,
virtualenvcan help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).
That's why people consider insert(0, to be wrong -- it's an incomplete, stopgap solution to the problem of managing multiple environments.