How do you use pip, virtualenv and fabric to deploy?

Question

What are your settings, your tricks and, above all, your workflow?

These tools are great, but there are still no best practices for using them, so I don’t know what is the most effective way to use them.

• Do you use pip bundles or always download?
• Do you configure Apache / Cherokee / MySQL manually or do you have a script for this?
• Do you put everything in virtualenv and use --no-site-packages ?
• Are you using one virtualenv for multiple projects?
• What are you using Fabric for (what part of your deployment are you scripting)?
• Do you put your Fabric scripts on a client or server?
• How do you handle the migration of database files and multimedia files?
• Do you need a build tool like SCons ?
• What are the deployment steps? How often do you do each of them?
• and etc.

Answer 1

"Best practices" are very context sensitive, so I will not argue that my methods are better, only that they work for me. I work mainly on small sites, so there are no deployments with multiple servers, CDNs, etc. I need to support the deployment of Webfaction shared hosting, as some customers need the cheapest hosting that they can find. I often have to deploy sites several times in different environments, so recurring deployment scenarios are crucial.

• I do not use pip packages, I install from the .txt requirement. I start my own chishop server with sdists of everything I need, so there aren’t several points of failure during the build process, I also use PIP_DOWNLOAD_CACHE on my development machines to speed up the project’s boot environments, as most of my project requirements overlap a bit.
• I have Fabric Scripts that can automatically configure and configure nginx + Apache / mod_wsgi on Ubuntu VPS or configure the equivalent on Webfaction and then deploy the project.
• I do not use -no-site-packages with virtualenv because I prefer to have slow compiled packages (Python Imaging Library, psycopg2) installed at the system level; too slow and troublesome to do inside each virtual. I had no problems with contaminated system package sites, because I do not pollute it at all. And in any case, you can install another version of something in virtualenv, and it will take precedence.
• Each project has its own virtualenv. I have several bash scripts (not virtualenvwrapper , although many people use it and love it) that automate the deployment of virtualenv for a given project to a known location and setting the project requirements to it.
• The entire deployment process with an open Ubuntu VPS or Webfaction host account on a website works using Fabric.
• Fabric scripts are part of the project source tree, and I run them from a local development check.
• I do not need SCons (what I know).

Deployment

Currently, the new deployment is divided into the following stages:

• fab staging bootstrap (server installation and source deployment)
• fab staging enable (enable the Apache / nginx configuration for this site)
• fab staging reload_server (reload Apache / nginx configuration).

You can, of course, combine fab staging bootstrap enable reload_server into one command line.

Once these steps are completed, updating the deployment with the new code will be just fab staging deploy .

If I need to cancel the update, fab staging rollback . Nothing particularly magical in the rollback; it just rolls back the code to the last deployed version and transfers the database to a previous state (this requires some metadata to be recorded about the migration state after the database is deployed, I just do it in a text file).

Examples

I have not used the Fabric scripts described in this answer for several years, so they are not supported at all, and I disclaim responsibility for their quality :-) But you can see them at https://bitbucket.org/carljm/django -project-template - in fabfile.py in the root of the repo and in the deploy/ subdirectory.

Answer 2

I use the fabric to create and deploy my code and assume that the system is already set up for this. I think a tool like puppet is more appropriate for automating the installation of things like apache and mysql, although I have yet to include this in my workflow.

In addition, I usually have different virtualenv for each project. They are created from the "base" python installation, where, as Carl pointed out, you can leave some global python libraries.

So, in terms of workflow:

• to install the necessary services (web server, database, ssh server, ...)
• puppet to configure the necessary users and base folders
• to create virtualenv for the application
• Fabric to install the package from the requirement. txt
• fabric to deploy your application
• fabric for deploying configuration files (web server, ...)