High memory usage only with multiprocessing

Question

I am trying to use the python multiprocessing library to hope to get some performance. In particular, I use the map function . Now, for some reason, when I change it with a single processed instance, I don't get a lot of memory usage. But using the multiprocessor version of the card makes my memory go through the roof. For the record, I am doing something that can easily raise the load of memory, but what is the difference between them to cause such a sharp difference?

Answer 1

You understand that multiprocessing does not use threads, right? I say this because you mention the "single-threaded analogue."

Do you send a lot of data through multiprocessing map? The likely reason is the multiprocessing multitasking, which should be performed in many cases. multiprocessinguses picklethat usually takes up more memory than the data it poisons. (In some cases, especially on systems with fork()where new processes are created when the method is called map, this can avoid serialization, but whenever it needs to send new data to an existing process, it cannot do this.)

multiprocessing , , . , , . ( , ) CoW, Python , , , .