Will Python use all processors in thread mode?

Question

When developing a Django application deployed on Apache mod_wsgi, I found that in the case of multithreading (Python threads, mod_wsgi processes = 1 threads = 8), Python will not use all available processors. Thanks to the multiprocessor approach (mod_wsgi processes = 8 threads = 1) everything is in order, and I can fully load my machine.

So the question is: is this Python behavior normal? I doubt it because using 1 process with multiple threads is the standard mod_wsgi method.

System:

2xIntel Xeon 5XXX series (8 cores (16 with hyper-thread)) on FreeBSD 7.2 AMD64 and Python 2.6.4

---

Thanks to everyone for the answers. We all found that this behavior is normal due to the GIL. Here is a good explanation: http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/ or stackoverflow. GIL talk: What is global interpreter lock (GIL)? .

Answer 1

Will Python use all processors in thread mode? Not.

Python will not use all available processors; Is this Python behavior normal? Yes, this is normal due to the GIL.

For discussion, see http://mail.python.org/pipermail/python-3000/2007-May/007414.html .

You may find that having a pair (or 4) threads per core / process can improve performance if there is some lock, for example, waiting for a response from the database will cause this process to block other connections otherwise.

Answer 2

• Will python use all processors in thread mode? Not.

• This is normal? Yes, it is normal. Python makes no effort to find all of your cores.

• "1 process with multiple threads is the default approach of mod_wsgi by default." But this is not optimal or even desirable. This is just a default. Do not read anything in it.

If you want to use all the resources of your computer, make the OS processing. Use processes.

The distinction between multiprocessor and multithreaded processing is difficult to measure for the most part. Using processes or threads hardly matters. It is usually easier to use processes, because there is trivial OS support for this.

Bottom row

Use several processes that allow the OS (and Apache) to make the most of the system.

Threads use a limited set of I / O resources for the Process in which they enter, and web page maintenance uses I / O binding. Processes have independent I / O resources and will make your processor easier to output.

Answer 3

There is still hope. GIL is just an artifact of the Python C implementation implementation that you download from python.org. Jython and IronPython are two other implementations of Python, and they do not have GIL, so you can have better results with one of them.

Answer 4

I do not know if this is still the case, but there is a global lock in the Python interpreter that prevents the use of all processor resources from one interpreter even when using multithreading. IIRC , global blocking associated with I / O.

It seems you are looking at the result of this lock, so personally, I would use several processes with one thread.

Answer 5

Yes. Python is not very multithreaded. Instead, there is a global lock, and each thread receives several operations in turn. This makes it easy to write MT applications in Python since there can be no problems with obsolete caches, etc.

Thus, a single Python process can occupy only one processor. To make full use of the multi-core system, you must run several Python processes.