Is there a high level profiling module for Python?

Question

I want to profile my Python code. I know cProfile well, and I use it, but it's too low level. (For example, there’s not even an easy way to catch the return value from the function you are profiling.)

One of the things that I would like to do is: I want to use the function in my program and set its profiling on the fly during program launch.

For example, suppose I have a heavy_func function in my program. I want to run the program and not use the heavy_func function. But sometime during the execution of my program I want to change heavy_func to the profile itself while it works. (If you are interested in how I can manipulate files while the program is running: I can do this either from a debug probe or from a shell integrated into my GUI application.)

Is there a module already written that does such things? I can write it myself, but I just wanted to ask, so I will not reinvent the wheel.

Answer 1

I wrote my own module for myself. I called it cute_profile . Here is the code . Here are the tests .

Here is a blog post explaining how to use it.

This is part of GarlicSim , so if you want to use it, you can install garlicsim and make from garlicsim.general_misc import cute_profile .

If you want to use it in Python 3 code, just install the Python 3 garlicsim .

Here's an outdated code excerpt:

 import functools from garlicsim.general_misc import decorator_tools from . import base_profile def profile_ready(condition=None, off_after=True, sort=2): ''' Decorator for setting a function to be ready for profiling. For example: @profile_ready() def f(x, y): do_something_long_and_complicated() The advantages of this over regular `cProfile` are: 1. It doesn't interfere with the function return value. 2. You can set the function to be profiled *when* you want, on the fly. How can you set the function to be profiled? There are a few ways: You can set `f.profiling_on=True` for the function to be profiled on the next call. It will only be profiled once, unless you set `f.off_after=False`, and then it will be profiled every time until you set `f.profiling_on=False`. You can also set `f.condition`. You set it to a condition function taking as arguments the decorated function and any arguments (positional and keyword) that were given to the decorated function. If the condition function returns `True`, profiling will be on for this function call, `f.condition` will be reset to `None` afterwards, and profiling will be turned off afterwards as well. (Unless, again, `f.off_after` is set to `False`.) `sort` is an `int` specifying which column the results will be sorted by. ''' def decorator(function): def inner(function_, *args, **kwargs): if decorated_function.condition is not None: if decorated_function.condition is True or \ decorated_function.condition( decorated_function.original_function, *args, **kwargs ): decorated_function.profiling_on = True if decorated_function.profiling_on: if decorated_function.off_after: decorated_function.profiling_on = False decorated_function.condition = None # This line puts it in locals, weird: decorated_function.original_function base_profile.runctx( 'result = ' 'decorated_function.original_function(*args, **kwargs)', globals(), locals(), sort=decorated_function.sort ) return locals()['result'] else: # decorated_function.profiling_on is False return decorated_function.original_function(*args, **kwargs) decorated_function = decorator_tools.decorator(inner, function) decorated_function.original_function = function decorated_function.profiling_on = None decorated_function.condition = condition decorated_function.off_after = off_after decorated_function.sort = sort return decorated_function return decorator 

Answer 2

It may be a little crazy, but this method should help you find the bottlenecks, this is what you want to do. You are pretty sure which routine you want to focus on. If you need to concentrate in this routine, it will prove that you are right. If the real problem is somewhere else, it will show you where they are.

If you need a tedious list of reasons, see here .