Running Two Class Methods Simultaneously in Python

Question

I am sure that many similar questions have been asked before, but after reading many of them, I am still not very sure what I should do. So, I have a Python script to control some external tools (camera and power meter). I wrote a class for both tools, calling C functions in .dll files using ctypes. Now it looks something like this:

for i in range(10): power_reading = newport.get_reading(N=100,interval=1) # take power meter reading img = camera.capture(N=10) value = image_processing(img) # analyze the img (ndarray) to get some values results.append([power_reading,value]) # add both results to a list 

I want to start executing the first two lines at the same time. Both newport.get_reading and camera.capture take about 100 ms-1 to run (they will work at the same time if I select the correct arguments). I don't need them to start EXACTLY at the same time, but ideally the delay should be less than about 10-20% of the total execution time (so there is less than 0.2 s delay when each takes about 1 s for launch). From what I read, I can use the multiprocessing module. So I'm trying something like this based on this post :

 def p_get_reading(newport,N,interval,return_dict): reading = newport.get_reading(N,interval,return_dict) return_dict['power_meter'] = reading def p_capture(camera,N,return_dict): img = camera.capture(N) return_dict['image'] = img for i in range(10): manager = multiprocessing.Manager() return_dict = manager.dict() p = multiprocessing.Process(target=p_capture, args=(camera,10)) p.start() p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1)) p2.start() p.join() p2.join() print(return_dict) 

I have a few problems / questions:

• I need to get return values ​​from both function calls. Using my current method, return_dict only shows the record for capture_img , but not the power counter, why? He also read that I can use Queue , what is the best method for my current purpose?

• How can I find out if both functions actually run at the same time? I’m thinking about using the time module to record both the start and end time of both functions, perhaps using some wrapper function to log the time log, but will there be any restrictions on using multiprocessing ?

• I usually run my code on an IDE (spyder), and from what I read, I need to run on the command line to see the result (I have some print statements inside functions for debugging purposes). Can I still work in the IDE to work both functions simultaneously?

Answer 1

Using Lock can help with synchronization:

 import multiprocessing def p_get_reading(newport, N, interval, lock, return_dict): lock.acquire() lock.release() reading = newport.get_reading(N, interval) return_dict['power_meter'] = reading def p_capture(camera, N, lock, return_dict): lock.acquire() lock.release() img = camera.capture(N) return_dict['image'] = img if __name__ == "__main__": for i in range(10): manager = multiprocessing.Manager() return_dict = manager.dict() lock = multiprocessing.Lock() lock.acquire() p = multiprocessing.Process(target=p_capture, args=(camera,10,lock,return_dict)) p.start() p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1,lock,return_dict)) p2.start() lock.release() p.join() p2.join() print(return_dict) 

Now you can create two Process objects and start() ed in any order, since the main procedure has already acquired a lock. After the release, the two processes will fight among themselves to acquire and free the castle, and be ready almost simultaneously.

Also note the use of if __name__ == "__main__" , as this helps when multiprocessing creates new processes.

I have to say it seems like Lock abuse

Answer 2

The answer to your first question is simply no if you do it the usual way, but yes if you want it. No, because the target function cannot communicate with the spawning stream using return . One way to do this is to use the queue and wrapper functions as follows:

 from threading import Thread from Queue import Queue def p_get_reading(newport,N,interval,return_dict): reading = newport.get_reading(N,interval,return_dict) return_dict.update({'power_meter': reading}) return return_dict def p_capture(camera,N,return_dict): img = camera.capture(N) return_dict.update({'image': img}) return return_dict def wrapper1(func, arg1, arg2, queue): queue.put(func(arg1, arg2)) def wrapper2(func, arg1, arg2, arg3, queue): queue.put(func(arg1, arg2, arg3)) q = Queue() Thread(target=wrapper1, args=(p_capture, camera, 10 , q)).start() Thread(target=wrapper2, args=(p_get_reading, newport, 100, 1, q)).start() 

Now q contains the updated and returned dict from p_capture() and p_get_reading() .