Simultaneous modeling in python

Question

I have a large set of simple simulations that I need to run, and I wonder if they can be done at the same time. Let me describe the situation: I have 1000 cases of prevalence for 100 diseases, and 1000 nobody's corresponding disability weights for these diseases (how bad it is to have this disease on a scale of 0-1) for 20 age groups. The simulation I need to do is determine, given the many prevalences, how many people will have different combinations of diseases. Here's what the raw data for 10 diseases will look like:

from __future__ import division import numpy as np disease_number = np.array([1,2,3,4]*10) age = np.array([5, 10, 15, 20]*10) prevalence = np.random.uniform(0, 1, (40, 1000)) disability_weight = np.random.uniform(0,1,(40, 1000)) 

A simulation of one draw would look something like this: at the age of 5 draw 1.

 prev_draw1 = prevalence[age==5, 1] disability_draw1 = disability_weight[age==5, 1] simulation = np.random.binomial(1, prev_draw1, (100000, prev_draw1.shape[0]) 

Then, to calculate the weight of the disability associated with each disease, given the concomitant pathology of multiple diseases, I do the following: Set the denominator as the sum of the current disability weights and use the disability weight of the disease as a numerator. For illness 1:

 denom = np.sum(disability_draw1**simulaiton) denom[denom==1]=0 numerator = disability_draw1*simulation[:, 0] adjusted_dw = np.sum(numerator/denom) 

I would need this adjusted dw calculation separately for each disease. Is there a way for me to do these 1000 simulations at the same time? Any help is appreciated, and I'm pretty new to python, so more descriptions are very helpful.

Answer 1

If you have multiple processors / cores, you can take a look at the multiprocessing module.

Running 1000 simulations at the same time can be a little expensive. You should probably run the simulation at a speed of one per core at a time.

You can use the Queue module and work with the process pool.

Here is a mini-sample of how it might look (not tested):

 from multiprocessing import Process, Queue def run_simulation(simulations, results): while simulations.qsize() > 0: simulation_params = simulations.get() # run simulation results.put(simulation_result) simulations.task_done() if __name__ == '__main__': simulations_to_run = Queue() simulations_to_run.put({}) # simulation parameters go in this dict, add all simulations, one per line (could be done in a loop, with a list of dicts) results = Queue() for i in range(8): #number processes you want to run p = Process(target=run_simulation, args=(simulations_to_run, results)) p.start() simulations_to_run.join() # now, all results shoud be in the results Queue 

http://docs.python.org/library/multiprocessing.html