How to measure Django cache performance?

Question

I have a rather small (about 4.5 thousand pages per day) website running on Django, with PostgreSQL 8.3 as db.

I use the database both cache and sesssion server. I heard a lot of good things about how to use Memcached for this purpose, and I would definitely like to try. However, I would like to know exactly what the benefits of this change will be: I assume that my site may not be large enough to make better use of the backend for the cache. The fact is that I will not install and configure memcached, and I do not want to waste time trying to get nothing or a little.

How can I measure the overhead introduced with db as a cache backend? I looked at the django-debug toolbar, but if I understand correctly, you do not want to host the production site (you need to set DEBUG=True for it to work). Unfortunately, I can’t reproduce the settings on my laptop (I have a different OS, processor and a lot more RAM).

Does anyone compare different Django cache / session servers? Does anyone know what will be the difference in performance if I do, for example, one session record for each request?

Answer 1

In my previous work, we tried to measure the effect of caching on the site we were developing. On the same machine, we tested a set of 10 pages, which are most often used as start pages (lists of objects), as well as some pages of object details made randomly from a pool of ~ 200000. The difference was 150 requests / sec to 30,000 requests per second , and database queries dropped to 1-2 per page.

What was cached:

• session
• lists of objects received for each individual page in the list of objects
• secondary objects and shared content (found on each page)
• lists of object categories and other categorization properties
• object counters (calculated autonomously using the cron job)
• individual objects

In general, we used only low-level granular caching, and not a high-level cache structure. It required very careful design (the cache had to be invalid each time the state of the database changed, for example, adding or changing any object).

Answer 2

Short answer: if you have enougth ram, memcached will always be faster. You cannot compare memcached with the database cache, just keep in mind that the big bottleneck with the servers is disk access, especially write access.

In any case, disk cache is better if you have many objects for caching and long expiration. But for this situation, if you want to play concerts, it is better to create pages statically with the python script and deliver them using ligthtpd or nginx.

For memcached, you can configure the number of bars allocated to the server.

Answer 3

The DiskCache project publishes Django cache tests comparing local memory, Memcached, Redis, file and diskcache.DjangoCache . An additional advantage of DiskCache is the lack of a separate process (unlike Memcached and Redis). Instead, cache keys and small values ​​are mapped into the memory of the Django process. Retrieving values ​​from the cache is usually faster than memcached on localhost. A number of parameters determine how much data is stored in memory; the rest are uploaded to disk.

Answer 4

Just give it a try. Use firebug or a similar tool and run memcache with a small amount of RAM allocation (e.g. 64 MB) on the test server.

Mark your average boot results visible in firebug without memcache, then enable caching and mark the new results. It was as simple as said.

Results usually shock people because performance is very good.

Answer 5

Use django-debug-toolbar to find out how much time was saved in the SQL query