unfortunately, this module needs to be inside the package, and it also needs to be runnable as a script, sometimes. Any idea how I could achieve that?
It's quite common to have a layout like this...
main.py
mypackage/
__init__.py
mymodule.py
myothermodule.py
...with a mymodule.py like this...
#!/usr/bin/env python3
# Exported function
def as_int(a):
return int(a)
# Test function for module
def _test():
assert as_int('1') == 1
if __name__ == '__main__':
_test()
...a myothermodule.py like this...
#!/usr/bin/env python3
from .mymodule import as_int
# Exported function
def add(a, b):
return as_int(a) + as_int(b)
# Test function for module
def _test():
assert add('1', '1') == 2
if __name__ == '__main__':
_test()
...and a main.py like this...
#!/usr/bin/env python3
from mypackage.myothermodule import add
def main():
print(add('1', '1'))
if __name__ == '__main__':
main()
...which works fine when you run main.py or mypackage/mymodule.py, but fails with mypackage/myothermodule.py, due to the relative import...
from .mymodule import as_int
The way you're supposed to run it is...
python3 -m mypackage.myothermodule
...but it's somewhat verbose, and doesn't mix well with a shebang line like #!/usr/bin/env python3.
The simplest fix for this case, assuming the name mymodule is globally unique, would be to avoid using relative imports, and just use...
from mymodule import as_int
...although, if it's not unique, or your package structure is more complex, you'll need to include the directory containing your package directory in PYTHONPATH, and do it like this...
from mypackage.mymodule import as_int
...or if you want it to work "out of the box", you can frob the PYTHONPATH in code first with this...
import sys
import os
SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.dirname(SCRIPT_DIR))
from mypackage.mymodule import as_int
It's kind of a pain, but there's a clue as to why in an email written by a certain Guido van Rossum...
I'm -1 on this and on any other proposed twiddlings of the
__main__machinery. The only use case seems to be running scripts that happen to be living inside a module's directory, which I've always seen as an antipattern. To make me change my mind you'd have to convince me that it isn't.
Whether running scripts inside a package is an antipattern or not is subjective, but personally I find it really useful in a package I have which contains some custom wxPython widgets, so I can run the script for any of the source files to display a wx.Frame containing only that widget for testing purposes.
unfortunately, this module needs to be inside the package, and it also needs to be runnable as a script, sometimes. Any idea how I could achieve that?
It's quite common to have a layout like this...
main.py
mypackage/
__init__.py
mymodule.py
myothermodule.py
...with a mymodule.py like this...
#!/usr/bin/env python3
# Exported function
def as_int(a):
return int(a)
# Test function for module
def _test():
assert as_int('1') == 1
if __name__ == '__main__':
_test()
...a myothermodule.py like this...
#!/usr/bin/env python3
from .mymodule import as_int
# Exported function
def add(a, b):
return as_int(a) + as_int(b)
# Test function for module
def _test():
assert add('1', '1') == 2
if __name__ == '__main__':
_test()
...and a main.py like this...
#!/usr/bin/env python3
from mypackage.myothermodule import add
def main():
print(add('1', '1'))
if __name__ == '__main__':
main()
...which works fine when you run main.py or mypackage/mymodule.py, but fails with mypackage/myothermodule.py, due to the relative import...
from .mymodule import as_int
The way you're supposed to run it is...
python3 -m mypackage.myothermodule
...but it's somewhat verbose, and doesn't mix well with a shebang line like #!/usr/bin/env python3.
The simplest fix for this case, assuming the name mymodule is globally unique, would be to avoid using relative imports, and just use...
from mymodule import as_int
...although, if it's not unique, or your package structure is more complex, you'll need to include the directory containing your package directory in PYTHONPATH, and do it like this...
from mypackage.mymodule import as_int
...or if you want it to work "out of the box", you can frob the PYTHONPATH in code first with this...
import sys
import os
SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.dirname(SCRIPT_DIR))
from mypackage.mymodule import as_int
It's kind of a pain, but there's a clue as to why in an email written by a certain Guido van Rossum...
I'm -1 on this and on any other proposed twiddlings of the
__main__machinery. The only use case seems to be running scripts that happen to be living inside a module's directory, which I've always seen as an antipattern. To make me change my mind you'd have to convince me that it isn't.
Whether running scripts inside a package is an antipattern or not is subjective, but personally I find it really useful in a package I have which contains some custom wxPython widgets, so I can run the script for any of the source files to display a wx.Frame containing only that widget for testing purposes.
Explanation
From PEP 328
Relative imports use a module's __name__ attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to '__main__') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.
At some point PEP 338 conflicted with PEP 328:
... relative imports rely on __name__ to determine the current module's position in the package hierarchy. In a main module, the value of __name__ is always '__main__', so explicit relative imports will always fail (as they only work for a module inside a package)
and to address the issue, PEP 366 introduced the top level variable __package__:
By adding a new module level attribute, this PEP allows relative imports to work automatically if the module is executed using the -m switch. A small amount of boilerplate in the module itself will allow the relative imports to work when the file is executed by name. [...] When it [the attribute] is present, relative imports will be based on this attribute rather than the module __name__ attribute. [...] When the main module is specified by its filename, then the __package__ attribute will be set to None. [...] When the import system encounters an explicit relative import in a module without __package__ set (or with it set to None), it will calculate and store the correct value (__name__.rpartition('.')[0] for normal modules and __name__ for package initialisation modules)
(emphasis mine)
If the __name__ is '__main__', __name__.rpartition('.')[0] returns empty string. This is why there's empty string literal in the error description:
SystemError: Parent module '' not loaded, cannot perform relative import
The relevant part of the CPython's PyImport_ImportModuleLevelObject function:
if (PyDict_GetItem(interp->modules, package) == NULL) {
PyErr_Format(PyExc_SystemError,
"Parent module %R not loaded, cannot perform relative "
"import", package);
goto error;
}
CPython raises this exception if it was unable to find package (the name of the package) in interp->modules (accessible as sys.modules). Since sys.modules is "a dictionary that maps module names to modules which have already been loaded", it's now clear that the parent module must be explicitly absolute-imported before performing relative import.
Note: The patch from the issue 18018 has added another if block, which will be executed before the code above:
if (PyUnicode_CompareWithASCIIString(package, "") == 0) {
PyErr_SetString(PyExc_ImportError,
"attempted relative import with no known parent package");
goto error;
} /* else if (PyDict_GetItem(interp->modules, package) == NULL) {
...
*/
If package (same as above) is empty string, the error message will be
ImportError: attempted relative import with no known parent package
However, you will only see this in Python 3.6 or newer.
Solution #1: Run your script using -m
Consider a directory (which is a Python package):
.
βββ package
β βββ __init__.py
β βββ module.py
β βββ standalone.py
All of the files in package begin with the same 2 lines of code:
from pathlib import Path
print('Running' if __name__ == '__main__' else 'Importing', Path(__file__).resolve())
I'm including these two lines only to make the order of operations obvious. We can ignore them completely, since they don't affect the execution.
__init__.py and module.py contain only those two lines (i.e., they are effectively empty).
standalone.py additionally attempts to import module.py via relative import:
from . import module # explicit relative import
We're well aware that /path/to/python/interpreter package/standalone.py will fail. However, we can run the module with the -m command line option that will "search sys.path for the named module and execute its contents as the __main__ module":
vaultah@base:~$ python3 -i -m package.standalone
Importing /home/vaultah/package/__init__.py
Running /home/vaultah/package/standalone.py
Importing /home/vaultah/package/module.py
>>> __file__
'/home/vaultah/package/standalone.py'
>>> __package__
'package'
>>> # The __package__ has been correctly set and module.py has been imported.
... # What's inside sys.modules?
... import sys
>>> sys.modules['__main__']
<module 'package.standalone' from '/home/vaultah/package/standalone.py'>
>>> sys.modules['package.module']
<module 'package.module' from '/home/vaultah/package/module.py'>
>>> sys.modules['package']
<module 'package' from '/home/vaultah/package/__init__.py'>
-m does all the importing stuff for you and automatically sets __package__, but you can do that yourself in the
Solution #2: Set __package__ manually
Please treat it as a proof of concept rather than an actual solution. It isn't well-suited for use in real-world code.
PEP 366 has a workaround to this problem, however, it's incomplete, because setting __package__ alone is not enough. You're going to need to import at least N preceding packages in the module hierarchy, where N is the number of parent directories (relative to the directory of the script) that will be searched for the module being imported.
Thus,
Add the parent directory of the Nth predecessor of the current module to
sys.pathRemove the current file's directory from
sys.pathImport the parent module of the current module using its fully-qualified name
Set
__package__to the fully-qualified name from 2Perform the relative import
I'll borrow files from the Solution #1 and add some more subpackages:
package
βββ __init__.py
βββ module.py
βββ subpackage
βββ __init__.py
βββ subsubpackage
βββ __init__.py
βββ standalone.py
This time standalone.py will import module.py from the package package using the following relative import
from ... import module # N = 3
We'll need to precede that line with the boilerplate code, to make it work.
import sys
from pathlib import Path
if __name__ == '__main__' and __package__ is None:
file = Path(__file__).resolve()
parent, top = file.parent, file.parents[3]
sys.path.append(str(top))
try:
sys.path.remove(str(parent))
except ValueError: # Already removed
pass
import package.subpackage.subsubpackage
__package__ = 'package.subpackage.subsubpackage'
from ... import module # N = 3
It allows us to execute standalone.py by filename:
vaultah@base:~$ python3 package/subpackage/subsubpackage/standalone.py
Running /home/vaultah/package/subpackage/subsubpackage/standalone.py
Importing /home/vaultah/package/__init__.py
Importing /home/vaultah/package/subpackage/__init__.py
Importing /home/vaultah/package/subpackage/subsubpackage/__init__.py
Importing /home/vaultah/package/module.py
A more general solution wrapped in a function can be found here. Example usage:
if __name__ == '__main__' and __package__ is None:
import_parents(level=3) # N = 3
from ... import module
from ...module.submodule import thing
Solution #3: Use absolute imports and setuptools
The steps are -
Replace explicit relative imports with equivalent absolute imports
Install
packageto make it importable
For instance, the directory structure may be as follows
.
βββ project
β βββ package
β β βββ __init__.py
β β βββ module.py
β β βββ standalone.py
β βββ setup.py
where setup.py is
from setuptools import setup, find_packages
setup(
name = 'your_package_name',
packages = find_packages(),
)
The rest of the files were borrowed from the Solution #1.
Installation will allow you to import the package regardless of your working directory (assuming there'll be no naming issues).
We can modify standalone.py to use this advantage (step 1):
from package import module # absolute import
Change your working directory to project and run /path/to/python/interpreter setup.py install --user (--user installs the package in your site-packages directory) (step 2):
vaultah@base:~$ cd project
vaultah@base:~/project$ python3 setup.py install --user
Let's verify that it's now possible to run standalone.py as a script:
vaultah@base:~/project$ python3 -i package/standalone.py
Running /home/vaultah/project/package/standalone.py
Importing /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/__init__.py
Importing /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py
>>> module
<module 'package.module' from '/home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py'>
>>> import sys
>>> sys.modules['package']
<module 'package' from '/home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/__init__.py'>
>>> sys.modules['package.module']
<module 'package.module' from '/home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py'>
Note: If you decide to go down this route, you'd be better off using virtual environments to install packages in isolation.
Solution #4: Use absolute imports and some boilerplate code
Frankly, the installation is not necessary - you could add some boilerplate code to your script to make absolute imports work.
I'm going to borrow files from Solution #1 and change standalone.py:
Add the parent directory of package to
sys.pathbefore attempting to import anything from package using absolute imports:import sys from pathlib import Path # if you haven't already done so file = Path(__file__).resolve() parent, root = file.parent, file.parents[1] sys.path.append(str(root)) # Additionally remove the current file's directory from sys.path try: sys.path.remove(str(parent)) except ValueError: # Already removed passReplace the relative import by the absolute import:
from package import module # absolute import
standalone.py runs without problems:
vaultah@base:~$ python3 -i package/standalone.py
Running /home/vaultah/package/standalone.py
Importing /home/vaultah/package/__init__.py
Importing /home/vaultah/package/module.py
>>> module
<module 'package.module' from '/home/vaultah/package/module.py'>
>>> import sys
>>> sys.modules['package']
<module 'package' from '/home/vaultah/package/__init__.py'>
>>> sys.modules['package.module']
<module 'package.module' from '/home/vaultah/package/module.py'>
I feel that I should warn you: try not to do this, especially if your project has a complex structure.
As a side note, PEP 8 recommends the use of absolute imports, but states that in some scenarios explicit relative imports are acceptable:
Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages). [...] However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose.
Hi all!
Say I have this folderstructure:
package
βββ __init__.py
βββ module.py
βββ subpackage
βββ bar.py
βββ __init__.py
βββ subsubpackage
βββ __init__.py
βββ foo.pyAnd I would like to import a function called 'func' from 'bar.py' into 'foo.py. How would I do this?
I tried literally every combination possible...like:
from .bar import func
from subpackage.bar import func
from ..bar import func
from . import bar.func
Problem is....nothing works, as I always get this error:
ImportError: attempted relative import with no known parent package
When I read on stackoverflow, I read a lot of different answers, most of them involving manipulating sys.path and quite complex.
I dont really understand why such a simple thing would be so complicated...what am I missing here?
python - ImportError : Attempted relative import with no known parent package - Stack Overflow
Relative Imports
Attempted relative import with no known parent package
ImportError: attempted relative import with no known parent package
Why does attempted relative import with no known parent package happen?
Should I replace every relative import with an absolute import?
What does __package__ tell me?
It seems, from Python docs and experimenting, that relative imports (involving ., .. etc) only work if
- the importing module has a
__name__other than__main__, and further, - the
__name__of the importing module is pkg.module_name, i.e., it has to be imported from above in the directory hierarchy (to have a parent pkg as part of it's__name__.)
OR
the importing module is being specified via module syntax that includes a parent pkg as python -m pkg.module, in which case it's __name__ is still __main__, so it is being run as a script, yet relative imports will work. Here __package__ is set and used to find the parent package while __name__ is __main__; more here.
[After all that, it appears that __package__ and sys.path are key to determining if/how relative imports work. __name__ indicates script or module(i.e., __main__ or module_name). __package__ indicates where in the package the relative imports occur with respect to, and the top of __package__ needs to be in sys.path.]
So, continuing with @AmitTendulkar 's example, if you run this as > python main.py or > python -m main or > python -m ecommerce.products from the project root directory, or enter interactive python from that root directory and import main, or import ecommerce.products the relative imports in products.py will work.
But if you > python products.py or > python -m products from within ecommerce directory, or enter interactive python from that ecommerce directory and import products they will fail.
It is helpful to add
print("In module products __package__, __name__ ==", __package__, __name__)
etc. in each file to debug.
UPDATE:
How imports work depend on sys.path and __package__, not on __name__.
Issued from /home/jj, > python sub/mod.py has a sys.path, __package__ of /home/jj/sub, None -absolute imports of modules in sys.path work, relative imports fail.
> python -m sub.mod has sys.path, __package__ of /home/jj, sub -absolute imports of modules in sys.path work, relative imports work relative to sys.path + __package__.
It is more helpful to add
import sys
print("In module products sys.path[0], __package__ ==", sys.path[0], __package__)
etc. in each file to debug.
i had a similar issue on Windows, and what helped me was (adapted to your directory):
# local imports
import sys
sys.path.append(r"C:\path\to\your\project")
from ecommerce.database import Database
You should be running your project from the parent of project dir as:
$ python -m project.main # note no .py
This tells python that there is a package named project and inside it a module named main - then relative and absolute imports work correctly - once you change the import in main in either of
from .bar import Bar # relative
from project.bar import Bar # absolute
@kevin41's answer is solid. I wanted to add some additional info that seemed a bit much for comments.
As a general rule, your top level script, the one being run by Python and which has __name__=="__main__", should NOT be inside any package.
By running main.py inside of package, the package itself is never imported.
Relative imports are package relative, not path relative. Python knows it is working with a package import and should use relative imports based on the __package__ special variable being set, and __package__ gets set based on using an import with a ., such as import package.bar, in which case __package__=="package"
The __package__ special variable is never set for the top level script, which is why main.py must use absolute imports.
When main.py imports bar using an absolute import, the value of the __package__ special variable while bar is being processed is also None, because bar was not imported as part of a package, therefore it can't use relative imports either.
With some small changes we can observe how bar imports foo based on how main is importing bar.
bar.py
try:
from .foo import Foo
print("Using relative import")
except ImportError:
from foo import Foo
print("Using absoloute import")
print(f"{__package__=} {__name__=} {__file__=}")
class Bar():
@staticmethod
def get_foo():
return Foo()
Output when running main.py as you have it configured:
Using absoloute import
__package__='' __name__='bar' __file__='c:\\Users\\...\\package\\bar.py'
Moving main.py outside of the package directory and changing the import to from package.bar import Bar, gives us the output
Using relative import
__package__='package' __name__='package.bar' __file__='c:\\Users\\...\\package\\bar.py'
The correct solution is to move your top level script outside of the package folder. If your application needs a primary script within the package, then that script should use relative imports and it should be imported by another top-level script outside of the package.
Example structure:
project/
__init__.py
bar.py
foo.py
main.py
app.py
main.py first line:
from .bar import Bar
app.py
from package.main import main
main()
Additional relevant posts:
- Import from another file inside the same module and running from a main.py outside the module throws an import error
- relative python imports. What is the difference between single dot and no dot
I know I'm late to the party, but I think I've found a solution to this problem. Hopefully this will be useful to someone else working on a large Python project.
The trick is to try one import format and fall back to the other format if the first fails.
Approach 1
db.py
try:
# Assume we're a sub-module in a package.
from . import models
except ImportError:
# Apparently no higher-level package has been imported, fall back to a local import.
import models
On the plus side, this approach is pretty simple, but doesn't scale well (module names are duplicated). Scaling can be improved by importing programmatically.
Approach 2 (not recommended)
db.py
import importlib
root = 'project.modules.core'
my_modules = ['core', 'models']
for m in my_modules
try:
globals()[m] = importlib.import_module(root + '.' + m)
except ImportError:
globals()[m] = importlib.import_module(m)
globals() is the global symbol table.
Of course, now this functionality needs to be duplicated in every module. I'm not sure that's actually an improvement over the first approach. However, you can separate this logic out into its own independent package that lives somewhere on pythonpath.
Approach 3
package_importer.py
import importlib
def import_module(global_vars, root, modules):
for m in modules
try:
global_vars[m] = importlib.import_module(root + '.' + m)
except ImportError:
global_vars[m] = importlib.import_module(m)
db.py
import package_importer
root = 'project.modules.core'
my_modules = ['core', 'models']
package_importer.import_module(globals(), root, my_modules)
This may be a bit outdated, but maybe someone else will benefit of my answer. Since python2 und python3 have different default import behavior, you have to determine between these two python versions.
Python 2.X
The default behavior for import models is to look up first the relative and then the absolute search path order. Therefore it should work.
However, in Python 3.X the default behavior for import models is to look for the model only in the absolute paths (called absolute imports). The current package core gets skipped and since the module db cannot be found anywhere else in the sys.path search path, it throws an error. To resolve this issue you have to use the import statement with dots from . import models to make clear that you are trying to import from a relative directory.
If you are interested to learn more about importing python modules, I suggest you to start your research with the following key words: module search path, python package import and relative package imports