I am trying to use the python multiprocessing package to speed up some of the physical simulations that I do using multiple cores of my computer.
I noticed that when I run the simulation, no more than 3 of the 12 cores are used. In fact, when I run the simulation, it first uses 3 cores, and then after a while it goes to 1 core. Sometimes only one or two cores are used from the very beginning. I could not understand why (I do not change anything except closing several terminal windows (without any active processes)). (OS - Red Hat Enterprise Linux 6.0, Python version 2.6.5.)
I experimented by changing the number of pieces (from 2 to 120) into which the work is divided (i.e. the number of processes being created), but this does not seem to have an effect.
I searched the Internet for information about this problem and read most of the related questions on this site (for example, one , two ), but could not find a solution.
(Edit: I just tried running the code under Windows 7, and it used all the available kernels. I still want to fix this for RHEL.)
Here is my code (with physics turned off):
from multiprocessing import Queue, Process, current_process def f(q,start,end): #a dummy function to be passed as target to Process q.put(mc_sim(start,end)) def mc_sim(start,end): #this is where the 'physics' is p=current_process() print "starting", p.name, p.pid sum_=0 for i in xrange(start,end): sum_+=i print "exiting", p.name, p.pid return sum_ def main(): NP=0 #number of processes total_steps=10**8 chunk=total_steps/10 start=0 queue=Queue() subprocesses=[] while start<total_steps: p=Process(target=f,args=(queue,start,start+chunk)) NP+=1 print 'delegated %s:%s to subprocess %s' % (start, start+chunk, NP) p.start() start+=chunk subprocesses.append(p) total=0 for i in xrange(NP): total+=queue.get() print "total is", total #two lines for consistency check: # alt_total=mc_sim(0,total_steps) # print "alternative total is", alt_total while subprocesses: subprocesses.pop().join() if __name__=='__main__': main()
(Actually the code is based on the answer from Alex Martelli here .)
Edit 2: eventually the problem resolved without me, as I understand it. I have not changed the code and I do not know that I have changed anything related to the OS. Despite this, now all the kernels are used when I run the code. Perhaps the problem will appear again, but for now I decided not to investigate further, as this works. Thank you all for your help.