Python Long-Term Server Profiling

Question

I have a long-term twisted server.

In a large system test, at one particular moment, several minutes in the test, when some clients enter a certain state and some specific external event occurs, this server takes several minutes of a 100% processor and makes it work very slowly. I would like to know what he is doing.

How do you get a profile for a certain period of time on a long-term server?

Could I easily send server start and stop messages via HTTP if there was a way to enable or deploy the profiler at runtime?

Given the choice, I would like to build on the stack / profiling of the call graph, but even fetching the sheet could give an idea.

Answer 1

yappi profiler can be started and stopped at runtime.

Answer 2

Not really an answer to Pythonic, but maybe the strace process gives some idea (if you are on Linux or similar).

Using strictly Python, for such things, I use tracking all calls, saving their results to the clipboard and using a signal (maybe you can do this with your HTTP message) to flush this buffer. Of course, tracing slows everything down, but in your scenario you can enable tracing using an HTTP message, so it will only be enabled when your problem is also active.