Trying to figure out the best way to import custom modules into my main script. It works fine if I have them all in the same directory like this:

project/project/
---- __init__.py
---- project.py
---- module1.py
---- module2.py 

And then these import statements in project.py work fine:

from .module1 import ClassName1
from .module2 import ClassName2

But if I'm trying to organize my structure more and use subdirectories like this:

project/project/
---- __init__.py
---- project.py
---- subdir/
-------- __init__.py
-------- module1.py
-------- module2.py

And then I try to import these modules in project.py like this:

from subdir.module1 import ClassName1
from subdir.module2 import ClassName2

And I get an error of:

ImportError: No module named 'subdir'

But the most frustrating part is that if I cd into project/project/ and run the python shell the imports work:

$ python
>>> from subdir.module1 import ClassName1
>>> from subdir.module2 import ClassName2

No import errors. I keep finding threads on stackexchange that say all you have to do is add a blank __init__.py file to the subdirectory which I'm doing but I keep getting these Import errors. The fact that it works in the python shell is the worst part because I don't know why it works there but not in my actual script.

What am I doing wrong?

EDIT: I uploaded a example project to github that might be easier to show what I'm doing. In the project I have two lines for two custom modules:

from .subdir.hellosubdir import HelloSubdir
from .hellosamedir import HelloSamedir

As it is now it give the error:

from .subdir.hellosubdir import HelloSubdir
ImportError: No module named 'helloimports.subdir'

If I comment out that line the line from .subdir.hellosubdir import HelloSubdir the other line imports without any problems.

Here's the github link:

https://github.com/thetalentedmrpeanut/helloimports

Also I'm installing this via pip and running it via cli with the $ helloimports command

EDIT 2:

Ok so I think I'm starting to wrap my brain around this. Here's what I found after some more digging:

First I looked up exactly where pip is installing my projects after I run the command $ pip install --user projectname/. I did this by opening a python shell and runing the following commands:

>>> import sys
>>> sys.path

Because I'm installing these packages with the --user flag a per the instructions of my operating system (gentoo linux) these packages are being installed in /home/myusername/.local/lib64/python3.4/site-packages/

When I went into that directory and then into the helloimports directory I discovered that my subdir directory was not even being installed. I did some more research and I think a big part of my misunderstanding has to do with the terms and how python defines certain things. As I understand it now a PACKAGE is a directory that contains a __init__.py file, a MODULE is a file with a .py extension that contains python code. Then there's also PROJECT which I'm going to think of as the entire collection of files.

The confusing thing about PACKAGE is that it's different than how a linux distribution would define a package. So on my system the package that gets installed is what I refer to as the PROJECT.

So one issue was that in my mind placing a module into a sub directory was just a matter of keeping multiple modules organized. I wasn't thinking that the act of placing them inside a sub directory was in fact creating a PACKAGE. I thought that a sub directory with a blank __init.__py was the same thing as having that module file in the same directory.

So I needed to find out why subdir was not being installed and I found out that the problem was in my setup.py file. In that file I had the line packages=['helloimports'],

In my mind thinking that my subdir was part of the helloimports package I thought that it should just automatically be included. But now that I know that the act of creating a subdirectory creates package it makes sense that it would not be installed.

To install the subdir package I can either explicitly add like: package=['helloimports', 'subdir'] or I can import the find_packages function from setuptools and then change the line to package=find_packages() this will automatically add all the sub packages within the main package unless you pass an argument excluding certain ones etc.

So the summary for the issue of importing sub directories is:

1. if you place a module in a sub directory it is no longer just a module it is now a PACKAGE

2. This new sub directory PACKAGE must contain an __init__.py file. This file can be blank or it can have code in it depending on your preference.

3. In setup.py you must either explicity add the subdirectory name to the setup() package= argument or you must import the find_packages function with from setuptools import setup, find_packages and then set packages=find_packages()

4. In the main file for you PROJECT if you want to import the sub directory module you use the dot notation like: from .subdirectory.modulename import ClassName

Next I discovered that my bin/helloimports file was not even doing anything. I removed it completely from the project and and my commandline command of $ helloimports --print-hello-subdir still worked. I looked at what pip was actual installing in the directory .local/bin and that was the following code:

#!/usr/bin/python3.4
# EASY-INSTALL-ENTRY-SCRIPT: 'helloimports==0.1.0','console_scripts','helloimports'
__requires__ = 'helloimports==0.1.0'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('helloimports==0.1.0', 'console_scripts', 'helloimports')()
    )

This didn't look anything like the bin/helloimports file that I had written. My bin/helloimports script was:

#!/usr/bin/env python

import helloimports
helloimports.helloimports.main()

But as it turns out I could have written anything in that file because it wasn't even being used. Again this was an issue with my understanding of setup.py. I had previously been using the setup() parameter scripts= for my python projects projects. So in past 2.7 projects I was doing something like scripts=['bin/helloimports'], in setup() in setup.py.

In new python 3 projects I've still been creating these bin/projectname scripts but I've been using the entry_points= parameter. I for some reason was thinking that this was interacting with my bin/projectname file but I have learned that it is not. And I also learned that it's much more powerful than I thought and you can, I think, create command line executables that target just a single function within a module.

So when I had the following in setup() in setup.py:

entry_points={ 
    'console_scripts': [
        'helloimports=helloimports.helloimports:main',
    ],
},

It had nothing to do with my bin/helloimports file. Instead it's defining the cli executable as helloimports and setting that equal to main package name helloimports --> main module name helloimports.py --> function of main()

This also means that the code in my helloimports/__init__.py file wasn't really doing anything either. I had it as

helloimports/__init__.py:

import helloimports

I don't think this was doing anything at all. Or at least it wasn't doing what I thought it was. If I delete that line and have a completely blank __init__.py file the commandline executable still works because I'm explicity connecting the main module of helloimports.py in the chain.

So how I understand it now is that the line helloimports=helloimports.helloimports:main can be broken down into the following parts:

1. helloimports= is the commandline executable that is used when running $ helloimports --help etc.

2. helloimports. is the helloimports package which is the helloimports/__init__.py file.

3. .helloimports is the helloimports/helloimports.py file

4. .helloimports:main is the def main(): function in helloimports/helloimports.py

So the issue comes into how to deal with these __init__.py files. From my understanding I can do a couple of different things.

1. I can leave the __init__.py file blank. This creates the need for the chaining of the blank helloimports/__init__.py to the helloimports.py in the console_scripts in setup.py. helloimports.helloimports:main

2. I can place all the code for my main module helloimports.py in __init__.py. Then my project would change from:

helloimports/
---- __init__.py
---- helloimports.py

to:

helloimports/
----__init__.py

And this would change the console_scripts to `helloimports=helloimports:main'

3. I can add some code to __init__.py to import everything from the helloimports.py module into the __init__py file and in doing so let the main PACKAGE and main MODULE be the same thing in the eyes of python. So this would look like:

helloimports/
---- __init__.py
---- helloimports.py

And helloimports/__init__.py would have the line from .helloimports import *

This would also be a situation where console_scripts in setup.py looks like helloimports=helloimports:main

Personally I perfer to have my main module file named the same as the package and I don't like putting tons of code in __init__.py but I also see the argument that the execessive chaining with the same name can get confusing. So I think I will try the #3 approach for now unless someone can prove to me that placing all your code in __init__.py is better. I would much rather think of __init__.py as being a simple file that defines a package not as being the file that contains all the code. If i'm searching for the main code in a python project it makes more sense to me to open up a projectname.py file than a __init__.py file that could technically be blank.

tl;dr - It's working for me now, I am still potentially violating best practices but at least now I can directly break or fix the imports in a way that makes sense to me. I've uploaded the new version of the project as a separate repo here so that it's easy to see the changes between the 2 versions:

https://github.com/thetalentedmrpeanut/helloimports2