Why does Python’s multiprocessing Pool.map cause infinite recursive spawning of child processes on Windows?

by

I’m writing a script for batch image processing that uses multiprocessing.Pool to run tasks in parallel. It works fine on Linux, but when I copy it to Windows, it freezes immediately. The number of python.exe processes skyrockets in Task Manager until the system runs out of memory. I didn’t wrap the script entry point with the `if __name__ == ‘__main__’` guard. I suspect this is the cause, but I’m not sure about the underlying mechanism.
# Relevant code snippets, run results, error messages

import multiprocessing as mp

def work(x):
    return x * x

if __name__ == '__main__':
    with mp.Pool(4) as pool:
        print(pool.map(work, range(10)))

Without the `if __name__ == ‘__main__’:` guard, running the script on Windows leads to repeated console output, freezing and a sharp surge in process count, while everything works normally on Linux.

# My initial reasoning
I recall that Windows uses the `spawn` start method. Child processes re-import the main module. Without the `__main__` guard, the code that creates the Pool will execute again, spawning child processes recursively. However, I am not sure whether adding the guard alone makes it fully safe. I also wonder if there are additional pitfalls related to where the Pool is created and global variable initialization.

# Relevant information such as operating environment and software versions
Python 3.10, Windows 10, standard `multiprocessing` library (tested against Ubuntu 20.04 on Linux for comparison).

Expert Answer:

Your diagnosis is on the right track, but the mechanism deserves clarification: Windows does not support fork, so multiprocessing defaults to spawn. When a child process starts, it re-imports the main module. At this point, `__name__` is `__mp_main__` instead of `__main__`. As a result, the top-level `mp.Pool(4)` gets executed again. Every child process creates its own Pool, and each Pool spawns new child processes, leading to exponential process proliferation until memory is exhausted. On Linux, fork copies the parent process memory directly, and the child does not re-run top-level module code, so this issue does not appear.

Adding `if __name__ == ‘__main__’:` is a necessary condition, but not sufficient. Here are several common pitfalls:
The guard must wrap all code that triggers process creation, including the `pool.map` call itself. If you only put `Pool(…)` inside the guard block and leave `pool.map` outside, you will still encounter the problem.

Avoid side-effecting initialization at the module top level. Examples include top-level statements such as `df = pd.read_csv(…)`, loading large models, or opening database connections. Under spawn, every child process re-runs these steps, which is slow and may cause errors. This kind of initialization should be placed inside a function or passed to the Pool via the `initializer` parameter.

Global variables are not automatically synchronized across child processes. Under spawn, child processes get the module state after a fresh import. Changes made to global variables by the parent during runtime are invisible to child processes. Pass required values using `initializer`/`initargs` or `functools.partial`.

Worker functions must be pickleable. Functions defined at the module top level work fine. Nested functions or lambdas defined inside `__main__` will trigger a `PicklingError` under spawn.

When packaging to an executable with PyInstaller, you also need `mp.freeze_support()` at the entry point, otherwise infinite process spawning will occur.

Troubleshooting tips: Temporarily add `print(__name__)` at the module top level. On Windows, you will repeatedly see `__mp_main__` printed, which directly confirms repeated imports as the root cause. You can also check the active start method with `mp.get_start_method()`. If needed, explicitly call `mp.set_start_method(‘spawn’)` on Linux to reproduce the issue for local debugging.

Leave a Comment