Ubuntu + virtualenv = mess? virtualenv hates dist packages, wants package sites

Question

Can someone please explain to me what is happening with python in ubuntu 9.04?

I'm trying to deploy virtualenv , and the --no-site-packages flag doesn't seem to do anything with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I upgraded to setuptools 0.6c9 ) and everything seems to be installed on /usr/local/lib/python2.6/dist-packages

I assume that when installing a package using apt-get it is placed in /usr/lib/python2.6/dist-packages/ ?

The problem is that there is /usr/local/lib/python2.6/site-packages that just sits where it's empty. It would seem (looking at path in virtualenv ) that this is the virtualenv folder used as a backup. Thus, I even thought that I was --no-site-packages , I could not access my local system packages from any of my virtual servers.

So my questions are:

• How do I get virtualenv to point to one of dist-packages ?
• What dist packages should I point to? /usr/lib/python2.6/dist-packages or /usr/local/lib/python2.6/dist-packages/
• What is the meaning of /usr/lib/python2.6/site-packages ? There is nothing there!
• Does it first have to go first? If I have a newer version of the XYZ package installed in /usr/local/lib/python2.6/dist-packages/ and an older one (from ubuntu repos / apt-get) in /usr/lib/python2.6/dist-packages , which is imported when I import xyz ? I guess this is based on a list of paths, yes?
• Why the hell is this so confusing? Is something missing here?
• Where is it defined that easy_install should install on /usr/local/lib/python2.6/dist-packages ?
• Will this affect pip ?

Thanks to everyone who can clean this!

Answer 1

I will be tempted to crack it by making the site packages link to dist-packages, but I think this may affect other cases when you want to install some kind of extension other than ubuntu dist. I can not come up with a different answer to 1 except setting up virtual sources (since both ubuntu and virtualenv are so popular, I won’t be surprised if I find modified versions already exist).

Re 2, if you use / usr / local / bin / python, you should use the / usr / local version of lib (including site packages) and vice versa if you use / usr / bin / python.

Re 3, there will be something if you ever install the extension for / usr / bin / python from sources (not via easy_install or from the ubuntu distribution).

Re 4, yes, previous entries on the path take precedence.

Re 5, easy_install is easy only in its name - it does so much dark magic that it was carefully kept outside the python standard library, despite its convenience, because the consensus among us python commanders is that deep dark magic is for convenience "easy" only on the surface.

Re 6, I think the ubuntu change for easy_install is correct if it is, it is determined wherever canonical or other ubuntu supporters make their collective decisions.

Re 7, sorry, I don’t know. I don't have a reasonable recent ubuntu on hand to check.

Answer 2

I find that Mike Orr's answer from the virtual-env mailing list seems to be the best. Please note that the OP posted this question in both places.

Original mail content:

A few years ago, Debian created / usr / local / lib / pythonVERSION / site -packages, and compiled the Python binary to include it in the default search path. Ubuntu followed the Debian leader as usual. The Python developers did not like this because you came across a locally installed / usr / local / bin / python using the same directory site packages. Ubuntu finally decided to abandon the site packages and use dist-packages instead, the name they invented so that it would not interfere with anything. The history of the fault is somewhere there, if you google it, somewhere in Python an error tracker or SIG distutils or for example.

The system works, at least if you are using the Ubuntu virtual package. Some people have had problems using locally installed virtual Ubuntu because sys.path magic entries were not added or anything. I am not sure about the -no-site-packages because I never use this option: I run PIL and mysqldb from Ubuntu packages because it is sometimes difficult to compile their C dependencies. (Need right header files, Python ignores files headers, etc.)

So, Ubuntu Python packages are in / USR / Lib / pythonVERSION / distance-packages. Or that python support is for some reason. Locally installed Python packages are included in / usr / local / lib / pythonVERSION / dist -packages by default. Whenever I install the Ubuntu 9.04 system that I run:

$ sudo apt-get install python-setuptools (6.0c9) $ sudo apt-get install python-virtualenv (1.3.3) $ sudo easy_install pip $ sudo pip install virtualenvwrapper

Virtualenvs work fine, although I have not tried --no-site-packages.

I'm trying to deploy virtualenv, and the -no-site-packages flag doesn't seem to do anything with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I updated to setuptools 0.6c9)

These versions are like in Ubuntu 9.04, so you make it more difficult on your own by installing them locally.

and everything seems to be installed in / usr / local / lib / python 2.6 / dist-packages

Yes

I assume that when installing a package using apt-get it is placed in / usr / lib / python2.6 / dist-packages /?

Yes

4. Does it first have to go first? If I have a new version of the XYZ package installed in / usr / local / lib / python 2.6 / dist-packages / and older (from ubuntu repos / apt-get) in / usr / lib / python2.6 / dist-packages which is imported when i import xyz? I guess this is based on a list of paths, yes?

sys.path is scanned in order. The funny thing is that .pth eggs get sooner or later on the way than some people expect. But if you use pip for everything that it can do (for example, with the exception of installing pip itself, pre-compiled eggs and a snapshot of the local directory, which is a copy, not a link to an egg), you will not have many .pth eggs in any case.

5. Why the hell is this so confusing? Is there something I'm missing here?

This is poorly documented. I figured this out by browsing the web.

7. Will this affect the pip?

Yes, pip will automatically install / usr / local / library / pythonVERSION / site packages. Use "pip install -E $ VIRTUAL_ENV packagename" to install in virtualenv.

Answer 3

You really should not touch installing Ubuntu Python unless you are creating system administrative tools or creating something that could be considered a new system service.

If you use Ubuntu to develop or deploy Python applications, always create your own Python from the source, create it and use it for deployment. This way you will have all the directories in the right place, and virtualenv will work fine. If you deploy multiple Python applications to the server, make your Python in some place, such as /home/python or /opt/python , or somewhere outside your home directory. Make sure you have write permissions for the development group ( users ?) So people can easily add packages.

It also allows you to have two levels of packages. Those that are your own standard tools can be installed on your Python distribution and be part of the tarball you are deploying, and only application-specific packages will be in virtualenv.

Do not upgrade or modify the installed Ubuntu Python system.

Answer 4

Well, I have Ubuntu 9.04, and I quickly tried to set up a couple of sandboxes with site packages and one without it. And everything is working fine.

The only difference in the approach I used was the Ubuntu python-virtualenv package (1.3.3). And suppose it is configured by the Ubuntu team according to the settings of Ubuntu.

To summarize disabling easy_installed virtualenv for a while, use the batch python-virtualenv and make sure that it meets your expectations.

In fact, we use a similar installation for production without any problems. Alex had already answered the rest.

Answer 5

Another way to fix it:
stack overflow

Remember to do this in every virtual space where you need it, but don't rely on hacks or a special version of virtualenv