How long is the garbage collector used?

Question

My python program has an interesting performance behavior: the longer it runs, the slower it gets. Initially, it launches dozens of work units per minute. An hour later, so it takes tens of minutes per unit of work. My suspicion is that this is the cause of the overloaded garbage collector.

The trick is that my script is too hungry for cProfile to work in large runs. (see cProfile, memory consuming )

We created our own performance plugin, and we can observe most parts of our system, and none of them seem to be a problem. One rock that still isn't is GC.

Is there any other way (besides profile or cProfile) to find out how much time goes to the GC?

Answer 1

In Python, most garbage is collected using reference counting. One would expect it to be quick and painless, and it is unlikely that this is what you need. I assume that you are asking about the collector referenced by the gc module, which is used only for circular references.

There are a few things that may be helpful: http://docs.python.org/library/gc.html

Despite the fact that there is no direct method for collecting the garbage collector, you can turn it on and off, enable debugging, view the collection count, etc. All this can be useful in your search.

For example, on my gc system, the elapsed time is printed if you enable debug flags:

 In [1]: import gc In [2]: gc.set_debug(gc.DEBUG_STATS) In [3]: gc.collect() gc: collecting generation 2... gc: objects in each generation: 159 2655 7538 gc: done, 10 unreachable, 0 uncollectable, 0.0020s elapsed. 

All this aside, the first thing I would like to draw attention to is the evolution of the memory usage of your program as it starts. One possibility is that it simply reaches the limit of available physical memory and slows down due to excessive page crashes, and not because of anything related to the garbage collector.