Separation of django project development and production parts

Question

I am creating a website that relies on the result of a machine learning algorithm. All that is needed for the user part of the site is the output of the algorithm (class labels for a set of elements), which can be easily saved and retrieved from django models. The algorithm can be run once a day and not rely on user input.

Thus, this part of the site depends only on django and related packages.

But the development, configuration and evaluation of the algorithm uses many other python packages, such as scikit-learn , pandas , numpy , matplotlib , etc. It also requires the preservation of many different sets of class labels.

These dependencies cause some problems when deploying to heroku because numpy requires LAPACK/BLAS. . It also seems like it would be good practice to have as few dependencies as possible in a deployed application.

How can I separate the machine learning part from the user-facing part, but still complete it so that the results of the algorithm can be easily used?

I thought about creating two separate projects, and then somehow wrote to a user database, but it looks like this will lead to maintenance problems (dependency management, changes to database schemas, etc.).

As far as I understand, this problem is slightly different from using different settings or databases for production and development, since it is more related to managing different sets of dependencies.

Answer 1

Just move what we discussed with the answer, in case people have the same question, my suggestion is:

• Take some time to determine the dependencies of your site and the algorithm code.

• Dump the dependency list in requirements.txt for each project.

• Deploy them in different environments so conflicts do not run.

• Develop some API endpoints on the side of your site using the Django Rest Framework or Tastypie , and let your algorithm code update your model using the API. Use cron to regularly run algorithm code and push data.

Answer 2

Create a requirements file for each environment and a basic requirements file for these packages common to all environments.

  $ mkdir requirements $ pip freeze > requirements/base.txt $ echo "-r base.txt" > requirements/development.txt $ echo "-r base.txt" > requirements/production.txt 

Then adjust the design and production dependencies and install them in the right environment

 #change to your development virtualenv #$source .virtualenvs/development/bin/activate $ pip install -r requirements/development.txt #change to your production virtualenv #$source .virtualenvs/production/bin/activate $ pip install -r requirements/production.txt