Expand Python Search Path to Another Source

Question

I just joined a project with a fairly large existing code base. We are developing on Linux and do not use the IDE. We run the command line. I am trying to figure out how to get python to look for the correct path when starting project modules. For example, when I run something like:

python someprojectfile.py 

I get

 ImportError: no module named core.'somemodule' 

I get this for all my imports, I assume this is a problem with this path.

TL; DR:

How to get Python to search for ~/codez/project/ and all files and folders for * .py files during import operations.

Answer 1

There are several possible ways to do this:

• Set the PYTHONPATH environment variable to a colon-separated list of directories to search for imported modules.
• In your program, use sys.path.append('/path/to/search') to add the directory names that Python will look for to search for imported modules. sys.path is just a list of directories that Python searches for whenever it is prompted to import a module, and you can change it as needed (although I would not recommend deleting any of the standard directories!). All directories that you put in the PYTHONPATH environment variable will be inserted into sys.path when you start Python.
• Use site.addsitedir to add the directory to sys.path . The difference between this and simple additions is that when using addsitedir it also looks for .pth files in this directory and uses them to possibly add additional directories to sys.path depending on the contents of the files. See the documentation for more details.

Which one you want to use depends on your situation. Remember that when you distribute your project to other users, they usually install it in such a way that Python code files are automatically detected by the Python importer (that is, packages are usually installed in the site-packages directory), so if you sys.path with sys.path in your code, this may be unnecessary and may even have adverse effects when this code is executed on another computer. Regarding development, I would venture to suggest that setting up PYTHONPATH usually the best way.

However, when you use something that just runs on your own computer (or when you have non-standard settings, for example, sometimes in web application frameworks), it’s quite common to do something like

 import sys from os.path import dirname sys.path.append(dirname(__file__)) 

Answer 2

You should also read about python packages here: http://docs.python.org/tutorial/modules.html .

From your example, I would suggest that you really have a package in ~/codez/project . The __init__.py file in the python directory maps the directory to a namespace. If your subdirectories have a __init__.py file, you only need to add the base directory to your PYTHONPATH . For example:

PYTHONPATH = $ PYTHONPATH: $ HOME / adaifotis / project

In addition to testing the PYTHONPATH environment variable, as David explains, you can test it in python as follows:

 $ python >>> import project # should work if PYTHONPATH set >>> import sys >>> for line in sys.path: print line # print current python path 

...

Answer 3

I know that this branch is a little outdated, but it took me a while to understand its essence, so I wanted to share it.

In my project, I had the main script in the parent directory, and to differentiate the modules, I put all the supporting modules in a subfolder called "modules". In my main script, I import these modules as follows (for a module called report.py):

 from modules.report import report, reportError 

If I name my main script, this will work. HOWEVER, I wanted to test each module by including main() in them, and each of them called each, like:

 python modules/report.py 

Now Python complains that it cannot find a "module called modules". The key point here is that, by default, Python includes a script folder in its search path, but not CWD. So this error says, in fact: "I can not find the subfolder of the modules." This is because the subdirectory "modules" does not exist from the directory in which the report.py module is located.

I find the easiest solution for this is to add CWD to the Python search path by including it at the top:

 import sys sys.path.append(".") 

Now Python searches for CWD (current directory), finds the subfolder "modules", and all is well.

Answer 4

I read this question, looking for an answer, and he did not like it.

So, I wrote a quick and dirty solution. Just put this somewhere in your sys.path and it will add any directory under folder (from the current working directory) or under abspath :

 #using.py import sys, os.path def all_from(folder='', abspath=None): """add all dirs under `folder` to sys.path if any .py files are found. Use an abspath if you'd rather do it that way. Uses the current working directory as the location of using.py. Keep in mind that os.walk goes *all the way* down the directory tree. With that, try not to use this on something too close to '/' """ add = set(sys.path) if abspath is None: cwd = os.path.abspath(os.path.curdir) abspath = os.path.join(cwd, folder) for root, dirs, files in os.walk(abspath): for f in files: if f[-3:] in '.py': add.add(root) break for i in add: sys.path.append(i) >>> import using, sys, pprint >>> using.all_from('py') #if in ~, /home/user/py/ >>> pprint.pprint(sys.path) [ #that was easy ] 

And I like it because I may have a folder for some random tools and not have them part of the packages or something else, and still get access to some (or all) of them in a couple of lines of code.

Answer 5

The easiest way is to create the file "any_name.pth" and place it in the folder "\ Lib \ site-packages". You should find this folder wherever python is installed.

In this file, put the list of directories into which you want to save the modules for import. For example, create a line in this file as follows:

C: \ Users \ example ... \ example

You can say that this works by running this in python:

 import sys for line in sys: print line 

You will see that your catalog is printed, among others, from where you can also import. Now you can import the file "mymodule.py", which is located in this directory as easily as:

 import mymodule 

This will not import subfolders. To do this, you can imagine creating a python script to create a .pth file containing all the subfolders of the folder that you define. Perhaps it starts at startup.

Answer 6

Here is a great explanation of what is happening and how to fix it:

https://leemendelowitz.imtqy.com/blog/how-does-python-find-packages.html

Answer 7

New option for an old question.
Installing the fail2ban package on Debian seems to be hard-coded to install in /usr/lib/python3/dist-packages/fail2ban not in python3 sys.path .

 > python3 Python 3.7.3 (v3.7.3:ef4ec6ed12, Jun 25 2019, 18:51:50) [GCC 6.3.0 20170516] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages'] >>> 

so instead of just copying, I (bash) linked the library with newer versions.
Future updates to the original application will also automatically apply to related versions.

  if [ -d /usr/lib/python3/dist-packages/fail2ban ] then for d in /usr/lib/python3.* do [ -d ${d}/fail2ban ] || \ ln -vs /usr/lib/python3/dist-packages/fail2ban ${d}/ done fi