Reducing memory with multiprocessing?

Question

About 100 people work in one of my applications. It started as a threading application, but performance issues (latency) were affected. So I converted these workers to multiprocessing.Process es. The comparative example below shows that load reduction was achieved by increasing memory usage (factor 6).

So, where exactly does memory use come from if Linux uses a cow and workers don't share any data?

How to reduce the amount of memory? (Alternative question: how to reduce threading load?)

Benchmarks on Linux 2.6.26, 4 CPU 2G RAM: (Please note that CPU usage is% of a single processor, so full load is 400%. The numbers are from viewing Munin diagrams.)

  | threading | multiprocessing ------------------+-----------+---------------- memory usage | ~0.25GB | ~1.5GB context switches | ~1.5e4/s | ~5e2/s system cpu usage | ~30% | ~3% total cpu usage | ~100% | ~50% load avg | ~1.5 | ~0.7 

Background: The application processes events from the network and stores some of them in the MySQL database.

Answer 1

My understanding is that with dynamic languages ​​like Python, copy-by-copy is not as efficient as much more memory is written (and therefore copied) after forking. As the Python interpreter progresses through the program, much more is happening than just your code. For example, reference counting - a very object will be written too quickly, since reference counting should write a reference count into memory (start copy).

Given this, you should probably have a hybrid thread / processing approach. Have multiple processes to take advantage of multiple cores, etc., but each one runs multiple threads (so you can deal with the level of concurrency that you need). You just need to experiment with how many threads and processes you run.