Multiprocessing in Python

Python's multiprocessing module sidesteps the Global Interpreter Lock entirely. Instead of creating threads that share a single GIL, multiprocessing spawns entirely separate Python interpreter processes, each with its own memory space and its own GIL. These processes can run Python bytecode simultaneously on different CPU cores, achieving true parallel execution.

If you have read the comparison of threads, processes, and async, you know that multiprocessing is the correct choice for CPU-bound work where threading and asyncio cannot help. This article covers the practical API: creating and managing processes, using process pools for parallel map operations, and the key differences from threading that matter when you write real code.

The multiprocessing module intentionally mirrors the threading module's API. Where threading has Thread, multiprocessing has Process. Where threading has Lock, multiprocessing has Lock.

This design means you can often switch a CPU-bound program from threading to multiprocessing by changing the import and the class names. But the similarity is superficial. Processes do not share memory by default, cannot access each other's variables, and require serialization to pass data between them.

Understanding these differences prevents the most common multiprocessing bugs.

Creating and running a Process

The Process class works like Thread: you pass a target function and arguments, call start(), and wait with join(). The critical difference is that the function and its arguments must be picklable, because the child process receives them through serialization.

python

import multiprocessing
import os
 
def worker(task_id):
    print(f"Task {task_id} running in process {os.getpid()}")
 
if __name__ == "__main__":
    processes = []
    for i in range(4):
        p = multiprocessing.Process(target=worker, args=(i,))
        processes.append(p)
        p.start()
 
    for p in processes:
        p.join()
 
    print("All processes finished.")

Each of the four processes prints its own process ID, which is assigned by the operating system and differs from the parent's PID and from every other worker's PID:

text

Task 0 running in process 12345
Task 1 running in process 12346
Task 2 running in process 12347
Task 3 running in process 12348
All processes finished.

Each process gets a unique PID from the operating system. They run on separate CPU cores and execute in true parallel. The if __name__ == "__main__": guard is not optional on macOS and Windows.

Without it, the child process re-imports the module and re-executes the process creation code, spawning an infinite cascade of processes.

Processes have more startup overhead than threads. Creating a process takes tens to hundreds of milliseconds because the OS must duplicate or spawn a new interpreter. For short-lived tasks, the setup cost can dominate.

Use process pools to amortize this cost across many tasks.

ProcessPoolExecutor for parallel map

ProcessPoolExecutor from concurrent.futures is the simplest way to parallelize CPU-bound work. It maintains a pool of worker processes and distributes tasks among them. The API is identical to ThreadPoolExecutor.

python

from concurrent.futures import ProcessPoolExecutor
import math
 
def is_prime(n):
    if n < 2:
        return False
    for i in range(2, int(math.sqrt(n)) + 1):
        if n % i == 0:
            return False
    return True
 
if __name__ == "__main__":
    numbers = [10_000_000 + i for i in range(20)]
 
    with ProcessPoolExecutor() as executor:
        results = list(executor.map(is_prime, numbers))
 
    primes = [n for n, p in zip(numbers, results) if p]
    print(f"Found {len(primes)} primes")

The executor distributes the 20 prime checks across available CPU cores. Each worker process imports the module and executes is_prime independently. The results come back in the same order as the input numbers.

executor.submit() gives you Future objects for finer control:

python

from concurrent.futures import ProcessPoolExecutor, as_completed
 
if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        futures = {
            executor.submit(is_prime, n): n for n in numbers
        }
        for future in as_completed(futures):
            n = futures[future]
            if future.result():
                print(f"{n} is prime")

as_completed() yields futures in completion order, so you can process results as they become available rather than waiting for all tasks to finish.

multiprocessing.Pool for advanced patterns

multiprocessing.Pool predates ProcessPoolExecutor and offers features the executor does not: asynchronous map with callbacks, lazy imap, chunked mapping, and worker recycling with maxtasksperchild.

python

from multiprocessing import Pool
import time
 
def expensive_computation(x):
    time.sleep(0.5)
    return x * x
 
if __name__ == "__main__":
    with Pool(processes=4) as pool:
        # Synchronous map: blocks until all results are ready
        results = pool.map(expensive_computation, range(10))
        print(f"Synchronous: {results}")

The same pool can also run imap() for lazy, in-order results, and map_async() for a non-blocking call you poll or wait on later:

python

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        # imap: lazy iterator, yields results as they complete, in order
        for result in pool.imap(expensive_computation, range(5)):
            print(f"Got: {result}")
 
        # map_async: returns immediately with an AsyncResult object
        async_result = pool.map_async(expensive_computation, range(8))
        async_result.wait(timeout=10)
        print(f"Async: {async_result.get()}")

map() blocks until all results are ready. imap() returns an iterator that yields results as they become available, preserving input order, which lets you start processing early results before later tasks complete. map_async() returns immediately with an AsyncResult you can poll, wait on, or attach callbacks to.

starmap() unpacks argument tuples, useful when your function takes multiple arguments:

python

def add(a, b):
    return a + b
 
if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.starmap(add, [(1, 2), (3, 4), (5, 6)])
    print(results)

starmap() unpacks each tuple into positional arguments before calling add, so (1, 2) becomes add(1, 2) rather than a single two-element argument:

text

[3, 7, 11]

Chunking: tuning map performance

Both Pool.map() and executor.map() divide the input iterable into chunks and send one chunk to each worker process. The chunk size affects performance because serializing and sending each chunk has overhead. Too many small chunks waste time on IPC.

Too few large chunks leave some workers idle while others finish their last chunk.

Pool.map() accepts a chunksize parameter. The default heuristic works well for many tasks, but tuning it can improve throughput for workloads where individual task times vary widely.

python

import time
from multiprocessing import Pool
 
def fast_or_slow(x):
    time.sleep(0.1 if x % 10 != 0 else 0.5)
    return x * 2
 
if __name__ == "__main__":
    with Pool(processes=4) as pool:
        for chunksize in [1, 5, 10, 25]:
            start = time.perf_counter()
            pool.map(fast_or_slow, range(100), chunksize=chunksize)
            elapsed = time.perf_counter() - start
            print(f"chunksize={chunksize:>2}: {elapsed:.2f}s")

Smaller chunksizes give better load balancing when task durations vary. Larger chunksizes reduce IPC overhead when tasks are uniform. Benchmark your actual workload to find the right value.

When processes are better than pools

Pools are convenient but every task goes through serialization. For long-lived worker processes that maintain state across many work items, create Process objects directly and communicate through multiprocessing.Queue or multiprocessing.Pipe.

python

import multiprocessing
 
def worker(in_queue, out_queue):
    while True:
        item = in_queue.get()
        if item is None:
            break
        result = item * item
        out_queue.put(result)

Each of the four long-lived worker processes pulls items from the same input queue and writes results to the same output queue, so the main process just needs to feed work in and collect results out without tracking which worker handled which item:

python

if __name__ == "__main__":
    in_q = multiprocessing.Queue()
    out_q = multiprocessing.Queue()
 
    processes = [
        multiprocessing.Process(target=worker, args=(in_q, out_q))
        for _ in range(4)
    ]
 
    for p in processes:
        p.start()
    for i in range(20):
        in_q.put(i)
    for _ in processes:
        in_q.put(None)
    results = []
    for _ in range(20):
        results.append(out_q.get())
    for p in processes:
        p.join()
    print(f"Results: {results}")

This pattern keeps worker processes alive across many tasks, avoiding the startup cost of creating and destroying processes for each batch of work. It also lets workers accumulate state, like loading a large model or opening a persistent database connection, that would be recreated for every task in a pool.

Platform differences: fork vs spawn vs forkserver

Python's multiprocessing module supports three start methods that control how child processes are created:

• fork (Unix only, default on Linux): The child process is a complete copy of the parent's memory at the moment of fork. Fast but can cause issues with multithreaded parent processes.

• spawn (default on macOS and Windows since Python 3.8): The child process starts a fresh Python interpreter and imports the main module. Slower but safe and consistent across platforms. Requires picklable functions defined at module level.

• forkserver (Unix only): A server process is forked once at startup, and all subsequent child processes are forked from it. Reduces the memory duplication cost of repeated forks.

You can check and set the start method, but it must be set before creating any processes:

python

import multiprocessing
 
print(multiprocessing.get_start_method())
 
# Set the start method (call once, before any Process/Pool creation)
# multiprocessing.set_start_method("spawn")

For cross-platform code, write with spawn in mind: define worker functions at module level, use if name == "main":, and avoid relying on inherited memory state from the parent.

Process synchronization primitives

The multiprocessing module provides the same synchronization primitives as threading: Lock, RLock, Event, Condition, Semaphore, and Barrier. They work identically in API but coordinate across processes rather than threads.

python

import multiprocessing
 
def worker(lock, counter, iterations):
    for _ in range(iterations):
        with lock:
            counter.value += 1
 
if __name__ == "__main__":
    counter = multiprocessing.Value("i", 0)
    lock = multiprocessing.Lock()
 
    processes = [
        multiprocessing.Process(target=worker, args=(lock, counter, 1000))
        for _ in range(4)
    ]
 
    for p in processes:
        p.start()
    for p in processes:
        p.join()
 
    print(f"Counter: {counter.value}")

multiprocessing.Value and multiprocessing.Array create shared memory blocks accessible across processes. The "i" argument specifies the C type (signed integer). The lock serializes access so the increments are safe.

Without a lock, the same race condition that affects threads also affects processes sharing the same memory.

The next article covers sharing data between processes in detail, including Manager for sharing arbitrary Python objects, Pipe for bidirectional communication, and when to use each approach.

Practical multiprocessing checklist

Before writing multiprocessing code, verify that your workload is actually CPU-bound. Profile with cProfile or measure with time.perf_counter(). If the program spends most of its time in I/O, use threading or asyncio instead.

Always guard the entry point with if name == "main":. This is not optional on macOS and Windows. Put it at the top level, not inside a function.

Define worker functions at module level. Nested functions and lambdas are not picklable with the spawn start method. Write def my_func(x): at module scope, not lambda x: x * 2.

Limit worker processes to the number of available CPU cores. More processes than cores increases context switching without improving throughput. os.cpu_count() returns the number of logical cores.

Use ProcessPoolExecutor unless you need Pool-specific features. The executor's API is simpler, its error handling is cleaner, and it composes with ThreadPoolExecutor for mixed workloads.