Multiprocessing in python with more than two levels

Question

I want to make a program and want to make caviar like this process -> n process -> n process

Can a second-level process restart with multiprocessing? using python 2.6 multiprocessor module

Thnx

Answer 1

@vilalian answer is correct but concise. Of course, it is difficult to provide more information when your initial question was vague.

To expand a bit, you will have an original program that creates n processes, but they will be slightly different from the original in that you want them (each, if I understand your question) to create n more processes. This can be done either by running code similar to your original process, but generating new sets of programs that performed this task at hand, without further processing, or you could use the same code / input point by simply providing different arguments - something like

 def main(level): if level == 0: do_work else: for i in range(n): spawn_process_that_runs_main(level-1) 

and run it with level == 2

Answer 2

You can structure your application as a series of process pools exchanging through queues with any depth. Although it can get hairy pretty quickly (probably due to the need to switch context).

This is not erlang, although it is for sure.

Multiprocessing documents are extremely useful.

Here (a little too much to add a comment) is some code that I use to increase the bandwidth in a program that updates my channels. I have one survey of processes for feeds that need to be extracted, which leads to a queue of 4 working groups receiving these results and retrieving feeds, and the results (if any) are then queued for the Pool process to parse and put in line to return to the database. Executed sequentially, this process will be very slow due to the fact that some sites use their sweet time to react, so most of the time the process expected data from the Internet and would use only one core. Within this process-based model, I actually expect the most in the database, and my NIC is saturated most of the time, and all 4 cores actually do something. Your mileage may vary.

Answer 3

Yes - but you may run into a problem that would require a fix that I made yesterday in the python trunk. See Bug http://bugs.python.org/issue5313

Answer 4

Sure. Especially if you use fork to create child processes, they work like perfectly normal processes (like father). Flow control is completely different, but you can also use a subtype of the second level.

Please note that you do not complicate your program too much, as the sample program with two levels of flows is usually not used.