Django concurrency performance (apache2 prefork / mod_wsgi), what am I doing wrong?

Question

First of all, I am in no way dissatisfied with the work of my site running on Django, it does not receive massive traffic, just over 1000 visits per day.

I was curious how well it handled large traffic peaks, so I used the ab-tool for benchmarking.

I noticed that performance when concurrency is greater than 1 provides the same request volume as one simultaneous connection.

Shouldn't I increase the number of repetitions / s with concurrency?

Im in a virtual machine with 1 GB of RAM, apache2 (prefork), mod_wsgi, memcached and mysql.
All content on the page has been cached, the database does not accept any hits. And if memcached drops the record, then only 2 light (indexed) requests will be executed - and they should be immediately rewritten.

Benchmarking data: (note: I compared it with 2000 and 10k queries with the same results)

For the start page via apache2 / mod_wsgi via django:
-n100 -c4: http://dpaste.com/97999/ (58.2 reqs / s)

-n100 -c1: http://dpaste.com/97998/ (57.7 reqs / s)

For robots.txt, directly from apache2:
-n100 -c4: http://dpaste.com/97992/ (4917 reqs / s)
-n100 -c1: http://dpaste.com/97991/ (1412 reqs / s)

This is my apache conf: http://dpaste.com/97995/

Edit: added more information

wsgi.conf: http://dpaste.com/98461/

mysite.conf: http://dpaste.com/98462/

My wsgi handler:

import os, sys os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler() 

Answer 1

Since you are using prefork MPM and mod_wsgi in native mode with many processes, you may be killing the performance of your box. To get started, invite you to read:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

Using the built-in mode, like you, you must carefully configure the MPM settings. Setting MaxRequestsPerChild is not zero, it is not a good start, because you are going to periodically preempt Apache processes, as a result of which you will cause a surge in load, since everything should reboot.

Offers a working MPM and your Python web application running in mod_wsgi daemon mode. This to run will result in far fewer running processes, lower memory overhead, and more predictable system performance. Then you can study more closely why everything can work more slowly.

You should pay attention to what you get for the following output section "ab":

 Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 0 0 0.2 0 2 Waiting: 0 0 0.1 0 2 Total: 0 0 0.2 0 2 

If the max column shows large values, you get the cost of downloading the application due to the fact that you did not exclude them from your tests by preloading or by using a short restart interval.