Python Multiprocessing RuntimeError on Windows

Question

I have a class function (call it "alpha.py") that uses multiprocessing (processes = 2) to develop a process and is part of the Python package I wrote. In a separate Python script (let's call it "beta.py"), I created an object from this class and named the corresponding function that uses multiprocessing. Finally, all this is wrapped inside a Python script shell (let's call this "gamma.py"), which handles a lot of class objects and functions.

Essentially:

• Run it. /gamma.py from the command line
• gamma.py uses a subprocess and executes beta.py
• beta.py creates an instance of the object from the alpha.py class and calls a function that uses multiprocessing (processes = 2)

You have no problem with Mac or Linux. However, this becomes a problem on the Windows computer, and the error (and the documentation) suggests that I should write this somewhere:

if __name__ == '__main__': freeze_support() 

This other post also mentions the same thing.

However, I do not know exactly where these two lines should be. Currently, neither alpha.py, beta.py, nor gamma.py contains an if __name__ == '__main__': section. It would be great if someone could tell me where these two lines should go, as well as the rationale for this.

Answer 1

Actually, freeze_support() is not required here. You get a RuntimeError because you create and start your new processes at the top level of your beta module.

When a new process is created using multiprocessing on Windows, a new Python interpreter will be launched in this process and it will try to import the module with the target function to be executed. This is your beta module. Now, when you import it, all your top-level instructions should be followed, which will lead to the creation and start of a new process. And then, recursively, another process from that process, etc. Etc.

This is most likely not what you want, so new processes should be initialized and started only once when you start beta.py directly using subprocess .

if __name__ == '__main__': should be placed in beta.py, and then move the initialization and run the code for new processes in this section. After that, when beta.py is imported and does not start directly, the new process will not start, and you will not see any side effects.