Geek Answers Handbook

How can I profile the memory of a multi-threaded program in Python?

Is there a way to profile the memory of a multithreaded program in Python?

To profile the CPU, I use cProfile to create separate profilers profiles for each thread and then combine them. However, I could not find a way to do this using memory profiles. I am using heapy.

Is there a way to combine statistics in heapy like cProfile? Or what other memory profilers would you suggest is more suitable for this task.

A question was asked about profiling CPU usage in a multithreaded program: How to profile a multithreaded program in Python?

Another memory profiler question: Python memory profiler

+8
python multithreading profiling
source share
4 answers

If you are happy to profile objects and not raw memory, you can use the gc.get_objects() function so that you do not need to configure with a metaclass. In later versions of Python, sys.getsizeof() will also allow you to take a snapshot when determining which underlying memory these objects are using.

+7
source share

There are ways to get valgrind for the profile memory of python programs: http://www.python.org/dev/faq/#can-i-run-valgrind-against-python

+3
source share

Ok What I was exactly looking for does not seem to exist. So, I found a solution - a workaround for this problem.

Instead of profiling memory, I will profile objects. Thus, I can see how many objects exist at a certain time in the program. To achieve my goal, I used metaclasses with minimal modification to existing code.

The following metaclass adds a very simple routine to the __init__ and __del__ this class. The routine for __init__ increases the number of objects with this class name by one, and __del__ decreases by one.

 class ObjectProfilerMeta(type): #Just set metaclass of a class to ObjectProfilerMeta to profile object def __new__(cls, name, bases, attrs): if name.startswith('None'): return None if "__init__" in attrs: attrs["__init__"]=incAndCall(name,attrs["__init__"]) else: attrs["__init__"]=incAndCall(name,dummyFunction) if "__del__" in attrs: attrs["__del__"]=decAndCall(name,attrs["__del__"]) else: attrs["__del__"]=decAndCall(name,dummyFunction) return super(ObjectProfilerMeta, cls).__new__(cls, name, bases, attrs) def __init__(self, name, bases, attrs): super(ObjectProfilerMeta, self).__init__(name, bases, attrs) def __add__(self, other): class AutoClass(self, other): pass return AutoClass

The incAndCall and decAndCall functions use the global module variable that they exist.

 counter={} def incAndCall(name,func): if name not in counter: counter[name]=0 def f(*args,**kwargs): counter[name]+=1 func(*args,**kwargs) return f def decAndCall(name,func): if name not in counter: counter[name]=0 def f(*args,**kwargs): counter[name]-=1 func(*args,**kwargs) return f def dummyFunction(*args,**kwargs): pass

The mannequin function is just a very simple way. I am sure there are much better ways to do this.

Finally, whenever you want to see the number of objects that exist, you just need to look at the dictionary of counters. Example:

 >>> class A: __metaclass__=ObjectProfilerMeta def __init__(self): pass >>> class B: __metaclass__=ObjectProfilerMeta >>> l=[] >>> for i in range(117): l.append(A()) >>> for i in range(18): l.append(B()) >>> counter {'A': 117, 'B': 18} >>> l.pop(15) <__main__.A object at 0x01210CB0> >>> counter {'A': 116, 'B': 18} >>> l=[] >>> counter {'A': 0, 'B': 0}

Hope this helps you. That was enough for my case.

+1
source share

I used Yappi , with which I was successful in several special multi-threaded cases. He received excellent documentation, so you should not have too many configuration problems.

For memory-specific profiling, check out Heapy . Be careful, it can create some of the largest log files you have ever seen!

0
source share

More articles:


All Articles