Python DEAP genetic algorithm multi-core speed

Question

I am using Python DEAP pacakge and I want to use multi-core code, and I used the tutorial at http://deap.gel.ulaval.ca/doc/dev/tutorials/distribution.html to successfully do this using multiprocessing.

My question is this: using 8 cores, how much can I accelerate to theory? The reason I ask is because I want to decide how many people and generations I can run in the same amount of time as the single-line version. My code was used to run ~ 200 seconds and with 8 cores, now it takes ~ 0.5 seconds (this is 400x acceleration). Can I assume that something will be accelerated by 400X? I know this is difficult, but your help will be greatly appreciated.

In general, if anyone can help, I would like to understand how multicore is changing the flow of computing. Does this only mean a comparison of the scores of each person for different cores for each generation? Or does he run generations in parallel? If you know any documentation that I could read about this, let me know.

I did not provide sample code, because it is not needed, because it is a very high question.

Answer 1

Does this mean only a comparison of the estimates of each person for different cores for each generation or the parallel execution of generations?

The example displays the evaluate operation, so ...

 fitnesses = toolbox.map(toolbox.evaluate, invalid_ind) 

A single workflow hits the map: all invalid_ind are sorted in one queue, and when the kernel is available, the next person in the queue is assigned to this kernel to run the evaluate routine. When the queue is empty, all results are collected in a list and returned to fitnesses . From here the process continues on this lonely.

So:

• “Yes,” it displays each person’s score for different cores and
• "No," he does not run generations in parallel

At least this is what I guess when I asked this question . Depending on your application, of course, in my experience with DEAP and cProfile, the two best CPU time consumers rated people and copied.