Python manager.dict () is very slow compared to regular dict

Question

I have a file for storing objects:

jobs = {} job = Job() jobs[job.name] = job 

now I want to convert it to using the dict dispatcher because I want to use multiprocessing and have to share these dictated amont processes

 mgr = multiprocessing.Manager() jobs = mgr.dict() job = Job() jobs[job.name] = job 

just converting to use manager.dict (), everything worked out very slowly.

For example, if you use native dict, it took only 0.65 seconds to create 625 objects and save them in a dict.

The task itself takes 126 seconds!

Any optimization I can do to keep manager.dict () at the level with python {}?

Answer 1

The problem is that for some reason, each insert is pretty slow (117x slower on my machine), but if you update your manager.dict() with a regular dict, it will be a one-time and quick operation.

 jobs = {} job = Job() jobs[job.name] = job # insert other jobs in the normal dictionary mgr = multiprocessing.Manager() mgr_jobs = mgr.dict() mgr_jobs.update(jobs) 

Then use the mgr_jobs variable.

Another option is to use the widely accepted multiprocessing.Queue class.

Answer 2

If you use mgr.dict() inside a loop in your pool. You can use a local normal dict to temporarily store the results, and then update mgr.dict() outside the loop, for example your_mgr_dict.update(local_dict)