Forcing a thread to block all other threads from executing

Question

UPDATE:

This answer claims that what I am trying to do has not been possible since April 2013. However, this seems to contradict what Alex Martelli says in the Python Cookbook (p. 624, 3rd ed.):

When PyGILState_Ensure () returns, it always ensures that the thread call has exclusive access to the Python interpreter. This is true even if the calling C code starts another thread that is unknown to the interpreter.

The docs also seem to suggest that GIL can be acquired, which will give me hope (except that I cannot call PyGILState_Ensure() from pure python code, and if I create a C extension to call it, I'm not sure how embed my memory_daemon() into it.

I may be reading the Python answer or cookbook and documents incorrectly.

ORIGINAL QUESTION:

I need this thread (from the threading module) to prevent any other thread from starting while a certain segment of its code is running. What is the easiest way to achieve this?

Obviously, it would be great to minimize code changes in other threads to avoid using C and Direct OS calls and make it cross-platform for windows and Linux. But realistic, I will be glad that I have any solution for my real environment (see below).

Environment:

• Cpython
• python 3.4 (but could upgrade to 3.5 if that helps)
• Ubuntu 14.04

Use Case:

For debugging purposes, I compute the memory used by all objects (as gc.get_objects() reports), and print the summary report on sys.stderr . I do this in a separate thread because I want this resume to be delivered asynchronously from other threads; I put time.sleep(10) at the end of the while True , which calculates the actual memory usage. However, the memory report thread takes some time to complete each report, and I do not want all the other threads to move forward until the memory calculation is complete (otherwise, the memory snapshot will be very difficult to interpret).

Example (to clarify the question):

 import threading as th import time def report_memory_consumption(): # go through `gc.get_objects()`, check their size and print a summary # takes ~5 min to run def memory_daemon(): while True: # all other threads should not do anything until this call is complete report_memory_consumption() # sleep for 10 sec, then update memory summary # this sleep is the only time when other threads should be executed time.sleep(10) def f1(): # do something, including calling many other functions # takes ~3 min to run def f2(): # do something, including calling many other functions # takes ~3 min to run def main(): t_mem = th.Thread(target = memory_daemon) t1 = th.Thread(target = f1) t2 = th.Thread(target = f2) t_mem.start() t1.start() t2.start() # requirement: no other thread is running while t_mem is not sleeping 

Answer 1

The Python cookbook is correct. You have exclusive access to the Python interpreter at the time PyGILState_Ensure() returns. Exceptional access means that you can safely call all the functions of CPython. And that means the current C thread is also the current active Python thread. If the current C thread did not have a corresponding Python thread before, PyGILState_Ensure() will automatically create it for you.

This state is after PyGILState_Ensure() . And you also have a GIL acquired at that moment.

However, when calling other CPython functions, such as PyEval_EvalCode() or any other, they can implicitly make the GIL be released meanwhile. For example, this is so if the Python time.sleep(0.1) operator is implicitly receiving a call somewhere. Although the GIL is freed from this topic, other Python threads may be running.

You have only a guarantee that when PyEval_EvalCode() (or any other CPython function that you called) returns, you will again have the same state as before, that is, you are on the same active Python thread, and you have GIL again.

---

About your original question: there is currently no way to achieve this, that is, invoke Python code and avoid the GIL being released as a result somewhere in between. And this is good, otherwise you can easily end up in dead ends, for example. unless you allow any other thread to release some kind of lock that is currently being held.

About how to implement your use case: the only real way to do this is in C. You would PyGILState_Ensure() to get the GIL. And at this point you should only call those CPython functions that cannot have the side effect of calling other Python code. Be very careful. Even PyObj_DecRef() can call __del__ . It would be best to avoid calling the CPython functions and manually moving the CPython objects. Note that you probably don't need to do this as hard as you have outlined: there is a basic CPython memory allocator, and I think you can just get the information from there.

Read here about memory management in CPython.

The related code is in pymem.h , obmalloc.c and pyarena.c . See the _PyObject_DebugMallocStats() Function, although it cannot be compiled into your CPython.

There is also a tracemalloc module , which, however, will add some overhead. Perhaps its basic C code ( _tracemalloc.c file) is useful, however, to understand the insides a bit better.

---

About sys.setswitchinterval(1000) : This is only for passing Python bytecode and processing it. This is basically the main CPython loop in PyEval_EvalFrameEx in the ceval.c file. There you will find the following part:

 if (_Py_atomic_load_relaxed(&gil_drop_request)) ... 

All logic with a switching interval is considered in the file ceval_gil.h .

Setting the maximum switching interval means that the main loop in PyEval_EvalFrameEx will not be interrupted for a longer time. This does not mean that there are no other features that the GIL could release in the meantime, and another thread could work.

PyEval_EvalFrameEx will execute Python bytecode. Suppose this calls time.sleep(1) . This will call the inline implementation of the C function. You will find this in time_sleep() in the timemodule.c file. If you follow this code, you will find the following:

 Py_BEGIN_ALLOW_THREADS err = select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &timeout); Py_END_ALLOW_THREADS 

Thus, GIL is being released in the meantime. Now, any other thread waiting for a GIL can pick it up and run other Python code.

Theoretically, you might think that if you set a high switching interval and you never call any Python code, which in turn can issue a GIL at some point, you will be safe. Please note that this is almost impossible. For example. GC will be called from time to time, and any __del__ some objects may have various side effects.

Answer 2

You must use thread locks to execute code synchronously between threads. The answer is somewhat correct, but I would like the locals to come back to check if you really have a castle.

Do not use variables as described in another answer to check for a lock. Variables can be damaged between multiple threads. Reentor locks were intended to solve this problem.

It is also not true in this code that the lock is released provided that the code between them does not throw an exception. therefore always in the context of with or try-catch-finally .

Here is a great article explaining Python synchronization and docs streams .

Edit: OP response response to Python attachment in C

You misunderstood what he said in the cookbook. PyGILState_Ensure returns a GIL if the GIL is available in the current python interpreter , but not in C threads that are unknown to the python interpreter.

You cannot force GIL from other threads in the current interpreter. Imagine if you were able, then basically you will cannibalize all other threads.

Answer 3

Python always runs one thread at a time due to a Global Interpreter lock. He does not do this when multiprocessing involved. You can see this answer to learn more about GIL in CPython.

Please note that the pseudo-code, as I do not know how you create streams / use them / what code you execute in streams.

 import threading, time l=threading.Lock() locked=False def worker(): l.acquire() locked=True #do something l.release() def test(): while locked: time.sleep(10) #do something threads = [] t = threading.Thread(target=worker) threads.append(t) t = threading.Thread(target=test) threads.append(t) for th in threads: th.start() for th in threads: th.join() 

Of course, it can be written better and can be optimized.

Answer 4

As a solution to the stop gap (for obvious reasons), the following worked for me:

 def report_memory_consumption(): sys.setswitchinterval(1000) # longer than the expected run time # go through `gc.get_objects()`, check their size and print a summary # takes ~5 min to run sys.setswitchinterval(0.005) # the default value 

If anyone has a better answer, post it.