I’m building a small tool for batch processing tasks. The main process spins up multiple workers using `multiprocessing.Pool`. I want each worker to write its processing results into a shared dictionary, so the main process can aggregate everything at the end. But after execution, when the main process prints this dictionary, it is still empty even though the child workers clearly wrote data into it. A regular `dict` works even worse — changes made by child processes are completely invisible to the main process.
Relevant code snippets, runtime output, and error messages
import multiprocessing as mp
def worker(shared, key):
shared[key] = key * 2
print(f"child set {key} -> {shared[key]}")
if __name__ == "__main__":
mgr = mp.Manager()
shared = mgr.dict()
with mp.Pool(4) as pool:
pool.starmap(worker, [(shared, i) for i in range(4)])
print("main sees:", shared)
The logs show every child process printed `child set …`, yet the final output from the main process is `main sees: {}`. There are no error messages; the main process simply fails to retrieve the data silently.
My initial troubleshooting thoughts
I suspect writes to the Manager proxy object are not synchronised back to the main process in time, or the proxy connection gets closed early when Pool workers exit. I also wondered whether an explicit flush or close call is required inside child processes. I am unsure whether the correct approach is to collect results via a Queue for later aggregation, or if there is a known pitfall when using Manager.
Environment, software versions and other relevant details
Python 3.10, Windows 10, multiprocessing default spawn start method.
This issue is quite typical on Windows with the spawn start method. The root cause is usually not Manager itself. Instead, when the proxy object is re-serialized and deserialized inside the child process, it connects to the wrong Manager instance — or more commonly, the worker is actually writing to a separate proxy.
Run two quick checks first:
Print `id(shared)` and `shared._token` inside the worker to verify whether the main process and child processes receive the same token. If the tokens differ, the child processes are not getting a proxy linked to the same Manager.
Replace `pool.starmap` with `pool.apply_async`, then manually call `pool.close()` and `pool.join()`, to rule out timing issues where the `with` block exits early and shuts down the Manager.
For your code snippet, the most likely pitfall is this: **the object created by `mp.Manager()` does not have an explicit reference retained, and when `with mp.Pool(4)` exits, the Pool terminates child processes. The Manager server process’s lifecycle depends on the main process.** Under spawn mode, even though the `mgr` variable exists, the `shared` proxy reconnects using the Manager’s address during deserialization in the child process. If Pool workers start before the Manager server is fully ready, writes go to a temporary connection. The recommended fix is:
if __name__ == "__main__":
mgr = mp.Manager()
shared = mgr.dict()
pool = mp.Pool(4)
try:
pool.starmap(worker, [(shared, i) for i in range(4)])
pool.close()
pool.join()
finally:
print("main sees:", dict(shared))
mgr.shutdown()
Key points: Call `close()` before `join()`. Do not use a `with` statement that makes the Pool terminate immediately after `starmap` returns. Use `dict(shared)` to force a proxy read and avoid fetching a locally cached copy.
Another easily overlooked pitfall: A single-key assignment such as `shared[key] = key * 2` on a Manager dict triggers two RPC calls. Internally, `__setitem__` may perform a read followed by a write, which can lose updates under concurrency. For batch workloads, it is recommended that workers return `(key, value)` tuples, and the main process aggregates results from the return value of `pool.starmap`. Alternatively, use `mp.Queue` for collection, which is more stable and faster than a Manager dict. Manager is suited for low-frequency shared state, not high-write caches.