Geek Answers Handbook

GIL profiling

Is there a way to profile a python process to use GIL ? Basically, I want to know what percentage of time the GIL holds . The process is single threaded.

My motivation is that I have code written in Cython that uses nogil. Ideally, I would like to run it in a multi-threaded process, but in order to find out if this might be a good idea, I need to know if the GIL is free for a significant amount of time.

I found this related question starting from 8 years ago. The only answer is no. Hopefully everything has changed since then.

+3
source share
3 answers

Completely by accident, I found a tool that does just that: gil_load .

It was published after I posted the question.

Well done, @chrisjbillington.

>>> import sys, math
>>> import gil_load
>>> gil_load.init()
>>> gil_load.start(output = sys.stdout)
>>> for x in range(1, 1000000000):
...     y = math.log(x**math.pi)
[2017-03-15 08:52:26]  GIL load: 0.98 (0.98, 0.98, 0.98)
[2017-03-15 08:52:32]  GIL load: 0.99 (0.99, 0.99, 0.99)
[2017-03-15 08:52:37]  GIL load: 0.99 (0.99, 0.99, 0.99)
[2017-03-15 08:52:43]  GIL load: 0.99 (0.99, 0.99, 0.99)
[2017-03-15 08:52:48]  GIL load: 1.00 (1.00, 1.00, 1.00)
[2017-03-15 08:52:52]  GIL load: 1.00 (1.00, 1.00, 1.00)
<...>

>>> import sys, math
>>> import gil_load
>>> gil_load.init()
>>> gil_load.start(output = sys.stdout)
>>> for x in range(1, 1000000000):
...     with open('/dev/null', 'a') as f:
...         print(math.log(x**math.pi), file=f)

[2017-03-15 08:53:59]  GIL load: 0.76 (0.76, 0.76, 0.76)
[2017-03-15 08:54:03]  GIL load: 0.77 (0.77, 0.77, 0.77)
[2017-03-15 08:54:09]  GIL load: 0.78 (0.78, 0.78, 0.78)
[2017-03-15 08:54:13]  GIL load: 0.80 (0.80, 0.80, 0.80)
[2017-03-15 08:54:19]  GIL load: 0.81 (0.81, 0.81, 0.81)
[2017-03-15 08:54:23]  GIL load: 0.81 (0.81, 0.81, 0.81)
[2017-03-15 08:54:28]  GIL load: 0.81 (0.81, 0.81, 0.81)
[2017-03-15 08:54:33]  GIL load: 0.80 (0.80, 0.80, 0.80)
<...>
+2
source

If you're curious how many times GIL is accepted, you can use gdb breakpoints. For example:

> cat gil_count_example.py
import sys
import threading
from threading import Thread

def worker():
    k=0
    for j in range(10000000):
        k+=j
    return

num_threads = int(sys.argv[1])
threads = []
for i in range(num_threads):
    t = Thread(target = worker)
    t.start()
    threads.append(t)

for t in threads:
    t.join()

To break 3.X on take_gil

> cgdb --args python3 gil_count_example.py 8
(gdb) b take_gil
(gdb) ignore 1 100000000
(gdb) r
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y 0x00007ffff7c85f10 in take_gil
                                                   at Python-3.4.3 / Python / ceval_gil.h: 208
        breakpoint already hit 1886 times

To break 2.X on PyThread_acquire_lock

> cgdb --args python2 gil_count_example.py 8
(gdb) b PyThread_acquire_lock
(gdb) ignore 1 100000000
(gdb) r
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
  1       breakpoint     keep y   0x00000039bacfd410 
        breakpoint already hit 1584561 times

, , https://github.com/knielsen/knielsen-pmp

 > ./get_stacktrace --max=100 --freq=10 `/sbin/pidof python2`
 ...
 292  71.92% sem_wait:PyThread_acquire_lock

.

> ./get_stacktrace --max=100 --freq=10 `/sbin/pidof python3`
...
557  77.68%  pthread_cond_timedwait:take_gil
0

.

, , . , , , GIL -, , . , .

IO/ , , , . , , , , , , IO/native- , , - .

Depending on your use case, multiprocessing may work for cases that are mostly processor related. Multiprocessing adds overhead, so this is usually suitable for CPU-bound tasks that persist for a relatively long time (several seconds or more).

-1
source

More articles:


All Articles