memory consumption when using imap_unordered in Python multiprocessing
A separate thread is spawned to ingest data into the pool, but it doesn’t stop to wait until the tasks are processed. Therefore, if task definitions consume substantial amount of memory it can lead to OOM kill.
If you’re using
multiprocessing.Pool
, consider upgrading toconcurrent.futures.process.ProcessPoolExecutor
, because it handles killed workers better. It doesn’t affect the problem described in this question.