How to properly terminate child processes with multiprocessing in python

Question

I have several callback functions, and I would like to start them as several processes and terminate them using the signal of the parent process.

My current way to do this is to create a generic c_bool with multiprocessing.Value and set it to True , and then distribute it to all my processes when they are created. My processes start a while loop using a common bool, for example:

 while myC_bool: ...keep running... 

Then I can simply switch bool to False from my parent process, and all child processes will complete their final loop and exit.

Many have told me, and I read in the documents that you should avoid using shared memory when using multiprocessing. I was told that the best way to avoid this is to demonize the process, give it a special signal handler and send it sigint / sigterm / etc ...

My question is that bool exclusively supports loop persistence and ONLY ever changes its value from my parent process and reads it from several child processes, a suitable solution for quickly and safely terminating all my child processes? I feel that there is less overhead for all children to just look at one common bool than to send x the number of sigints to them.

Will dismantle a better solution? If so, I would like some help to understand why.

Answer 1

There are many good reasons to go with your decision:

• Easier to think than signals.
• He had fewer cross-platform issues.
• You already have code that works this way.
• This makes it easy to add a “graceful shutdown” mechanism if you want in the future.

... etc.

Keep in mind that if you cannot prove to yourself that the multiprocessing and basic OS primitives on each platform you care about are guaranteed to work without synchronization here, you need to put Lock or something else around every access to the shared bool. This is not very difficult, but ... as soon as you do this, using, for example, Event without a common bool can be even easier.

In any case, if one of them were your cause, I would say, excellent, do it that way. But, according to your question, you really chose this because of performance:

I feel that for all children there is less overhead to just look at one common bool than to send x the number of pigments to them

If this is your reason, you are almost certainly mistaken. Children should look at the common bool (and acquire a common lock!) Every time through a certain cycle, while the signal needs to be sent only to each child once. Thus, your overhead will almost certainly be much higher.

But in fact, I can’t imagine the overhead of sending one signal to a child process or even capturing an interprocess lock once per cycle per process, somewhere near a bottleneck in any useful program, so ... why the overhead even is the question here first? Do what makes the best sense in the easiest way.

Answer 2

Since you are careful about who changes the shared variable, this should be good.

There are many different solutions. For example. use multiprocessing.Event and terminate the process when it is installed. Or using multiprocessing.Connection objects (from Pipe). The latter can be used for two-way communication between parents and children. As a stop signal for children, followed by parent confirmation.

Answer 3

People who tell you “don't do this” are wrong. A shared memory point is the exchange of memory between multiprocessor systems and what you do.

You have a solution that 1) is simple, and 2) works. The signal / daemon approach 1) is really cool and 2) is harder to code correctly and 3) much harder to understand.

The only error that I see in your approach is the possibility that the process can see an outdated copy of bool from the CPU cache and linger a bit on shutdown. There are ways to flush the cache to make sure that this does not happen, but you probably do not need it, because for most applications, flushing the cache often happens automatically.

Stand your ground.