Best practice for Python-then-profile-then-C design pattern?

A popular software development template is:

  • Cross out the logic and algorithms in Python.
  • Profile to find out where the slow bits are.
  • Replace anyone with C.
  • Ship code, which is the best compromise between high-level and fast.

I say popular just because I saw people talking about it as a great idea.

But are there large projects that really used this method? Preferably, free software projects so I can watch and see how they did it, and maybe learn some best practices.

+4
source share
4 answers

There are many ways people approach development.

Sometimes people follow your three steps and find that the slow bits are related to the environment, so re-writing Python to C does not solve the problem. This type of slowness can sometimes be solved on the system side, and sometimes it can be solved in Python using a different algorithm. For example, you can cache network responses so that you don’t get to the network every time, or in SQL you can offload work into “stored procedures that run on the server and reduce the size of the result set. Usually, when you have something that you need to rewrite it in C, the first thing you need to do is look for an existing library and just create a Python shell if you don’t already have it.

Often step 1 is to crowd out the application architecture, suspecting that a performance issue may occur in some area, then select the C library (possibly already wrapped for Python) and use it. Then step 2 simply confirms that there are no really big performance issues that need to be addressed.

I would say that it is better if a team with one or more experienced developers tries to predict performance bottlenecks and mitigate them with existing modules from the very beginning. If you are new to python, then your three-step process is completely believable, that is, get the build and test code, knowing that there is a profiler and the ability to use fast C modules if you need it. And then there is psyco and various tools to freeze the application into a binary executable.

An alternative approach to this, if you know that you will need to use some C or C ++ modules, is to start writing a C application from scratch, but embed Python to do most of the work. This works well for experienced C or C ++ developers, because they have a general idea of ​​the type of code that is tedious to do in C.

+3
source

I thought so too when I started using Python

I did step 3 twice (which I can remember) in 12 years. Not often enough to call it a design pattern. This is usually enough to wrap the existing C library. Usually, someone else wrote the shell.

+2
source

Step 3 is incorrect. In the modern world, more than half the time, “slow bits” are associated with an I / O or network or are limited by some other resource outside the process. Overwriting them into anything just introduces errors.

0
source

Mercurial is a good example: it is written in Python and C , and it is about as fast as git, which is mainly written in C.

0
source

All Articles