Share Data Between Processes in Python

When you spawn a new Python process with the multiprocessing module, it starts with its own memory space. A variable you set in the parent process is invisible to the child. A list you modify in a worker process is invisible to the parent.

This isolation is by design. It is what lets each process run independently on its own CPU core without the GIL. But real programs need processes to share data, and Python gives you several tools for that, each with different performance characteristics and complexity tradeoffs.

If you have worked through the multiprocessing basics, you know how to create processes and pools. This article covers the mechanisms for making those processes share information.

The tools form a spectrum from low-level shared memory to high-level managed objects. Value and Array wrap raw shared memory blocks for C types. Manager creates a server process that proxies Python objects.

Queue and Pipe pass messages between processes using serialization. shared_memory lets you create named memory blocks that any process can attach to. Choosing the right tool depends on what data you need to share, how fast it must be, and how much synchronization you are willing to manage yourself.

Understanding process memory isolation

Before you choose a sharing mechanism, you need to understand why processes do not share data by default. When a child process starts with the spawn method (default on macOS and Windows), Python launches a fresh interpreter that imports your module. The child has its own copy of every global variable, initialized from scratch.

Changes in one copy do not affect the other.

python

import multiprocessing
 
shared_list = []
 
def worker():
    shared_list.append("modified by worker")
    print(f"Worker sees: {shared_list}")
 
if __name__ == "__main__":
    p = multiprocessing.Process(target=worker)
    p.start()
    p.join()
    print(f"Parent sees: {shared_list}")

The worker's own copy of the list gains an entry, but the parent's copy of shared_list never changes, since spawning the child gave it an independent copy of the module's global state rather than a reference to the parent's memory:

text

Worker sees: ['modified by worker']
Parent sees: []

The worker process appended to its own copy of shared_list. The parent's list stayed empty. This is the correct and expected behaviour for separate processes.

To share data, you must use one of the mechanisms described below.

Value and Array: fast shared memory for C types

multiprocessing.Value and multiprocessing.Array allocate shared memory blocks that multiple processes can read and write. They wrap C types, not Python objects, so they are fast but limited to integers, floats, and raw byte arrays.

python

import multiprocessing
 
def increment(shared_counter, lock):
    for _ in range(1000):
        with lock:
            shared_counter.value += 1
 
if __name__ == "__main__":
    counter = multiprocessing.Value("i", 0)
    lock = multiprocessing.Lock()
 
    processes = [
        multiprocessing.Process(target=increment, args=(counter, lock))
        for _ in range(4)
    ]
 
    for p in processes:
        p.start()
    for p in processes:
        p.join()
 
    print(f"Counter: {counter.value}")

All four processes increment the same shared memory location 1,000 times each, and the lock ensures none of those 4,000 increments are lost to a race, so the final count matches the expected total exactly:

text

Counter: 4000

The "i" specifies a signed integer. Other type codes include "d" (double), "f" (float), "c" (char), and "b" (signed char). Value stores a single value.

Array stores a sequence of the same type.

python

import multiprocessing
 
def fill_array(shared_arr, start):
    for i in range(5):
        shared_arr[start + i] = (start + i) * 10
 
if __name__ == "__main__":
    arr = multiprocessing.Array("i", 10)
 
    p1 = multiprocessing.Process(target=fill_array, args=(arr, 0))
    p2 = multiprocessing.Process(target=fill_array, args=(arr, 5))
 
    p1.start()
    p2.start()
    p1.join()
    p2.join()
 
    print(f"Array: {list(arr)}")

The two processes write to disjoint halves of the same shared array, and because their index ranges never overlap, the results merge cleanly into one sequence without any coordination between them:

text

Array: [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]

Each process fills a different region of the array, so no lock is needed. When processes write to overlapping regions, or when one process reads while another writes, use a lock to prevent data corruption.

Value and Array accept an optional lock parameter. By default, they create a new RLock. Pass lock=False to disable locking and manage synchronization yourself.

Pass an existing Lock to share it.

Manager: sharing Python objects through a server process

multiprocessing.Manager creates a separate server process that holds Python objects and gives worker processes proxy objects that forward operations to the server through IPC. This lets you share lists, dictionaries, sets, and other Python objects as if they were local.

python

import multiprocessing
 
def worker(shared_dict, key, value):
    shared_dict[key] = value
 
if __name__ == "__main__":
    with multiprocessing.Manager() as manager:
        shared_dict = manager.dict()
 
        processes = [
            multiprocessing.Process(
                target=worker, args=(shared_dict, f"key-{i}", i * 10)
            )
            for i in range(4)
        ]
 
        for p in processes:
            p.start()
        for p in processes:
            p.join()
 
        print(f"Shared dict: {dict(shared_dict)}")

Each of the four worker processes writes its own key into the manager-backed dict, and because every write goes through the manager's server process rather than local memory, the parent sees all four entries once every worker has finished:

text

Shared dict: {'key-0': 0, 'key-1': 10, 'key-2': 20, 'key-3': 30}

The manager supports list, dict, Namespace, Lock, RLock, Semaphore, Condition, Event, Barrier, Queue, Value, and Array. Every operation on a managed object goes through the server process, so individual operations are atomic with respect to other processes using the same manager. However, compound operations like checking a value and then modifying it are not atomic and still need explicit locking.

python

# NOT atomic: another process could modify shared_list between these calls
if len(shared_list) > 0:
    item = shared_list.pop()
 
# Atomic alternative: use a managed lock
with managed_lock:
    if len(shared_list) > 0:
        item = shared_list.pop()

The convenience of managed objects comes at a performance cost. Every attribute access, method call, and item lookup on a managed object sends data to the manager server process and waits for a response. For high-throughput numeric data, Value and Array are much faster.

Queue and Pipe: message passing between processes

multiprocessing.Queue is a process-safe FIFO queue that uses serialization to pass Python objects between processes. It works like the queue.Queue covered in thread communication with queues, but uses pipes and locks internally to work safely across process boundaries instead of just threads.

python

import multiprocessing
 
def producer(q):
    for i in range(5):
        q.put(f"Item {i}")
 
def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print(f"Consumed: {item}")
 
if __name__ == "__main__":
    q = multiprocessing.Queue()
 
    p1 = multiprocessing.Process(target=producer, args=(q,))
    p2 = multiprocessing.Process(target=consumer, args=(q,))
 
    p1.start()
    p2.start()
    p1.join()
    q.put(None)
    p2.join()

multiprocessing.Pipe creates a pair of connection objects for bidirectional communication between two processes. It is faster than Queue for point-to-point communication but does not support multiple producers or consumers.

python

import multiprocessing
 
def worker(conn):
    conn.send("Hello from worker")
    msg = conn.recv()
    print(f"Worker received: {msg}")
    conn.close()
 
if __name__ == "__main__":
    parent_conn, child_conn = multiprocessing.Pipe()
 
    p = multiprocessing.Process(target=worker, args=(child_conn,))
    p.start()
 
    print(f"Parent received: {parent_conn.recv()}")
    parent_conn.send("Hello from parent")
 
    p.join()

The parent blocks on parent_conn.recv() until the worker sends its greeting, and the worker blocks on conn.recv() until the parent sends its own message back, so both sides end up printing what the other side sent:

text

Parent received: Hello from worker
Worker received: Hello from parent

Pipe() returns two connection objects, one for each endpoint. Both endpoints can send and receive. By default, the pipe is duplex (bidirectional).

Pass duplex=False to create a one-directional pipe where one end only sends and the other only receives.

Pipes are faster than queues for point-to-point communication because they avoid the locking and signaling overhead of a full queue implementation. Use a pipe when exactly two processes communicate. Use a queue when you need multiple producers, multiple consumers, or the task_done() and join() coordination pattern.

shared_memory: zero-copy buffers for large data

Python 3.8 introduced multiprocessing.shared_memory, which lets processes share raw memory buffers without serialization. This is useful for large binary data like NumPy arrays, image buffers, or serialized protocol buffers.

python

import multiprocessing
from multiprocessing import shared_memory
import numpy as np
 
def worker(shm_name, shape, dtype):
    existing_shm = shared_memory.SharedMemory(name=shm_name)
    arr = np.ndarray(shape, dtype=dtype, buffer=existing_shm.buf)
    arr[0] = 99
    existing_shm.close()

The worker attaches to the shared block by name rather than receiving the array itself, which is what avoids copying or pickling the data when the child process starts:

python

if __name__ == "__main__":
    arr = np.zeros(10, dtype=np.int32)
    shm = shared_memory.SharedMemory(create=True, size=arr.nbytes)
 
    shared_arr = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf)
    shared_arr[:] = arr[:]
 
    p = multiprocessing.Process(
        target=worker, args=(shm.name, arr.shape, arr.dtype)
    )
    p.start()
    p.join()
 
    print(f"Modified: {shared_arr}")
    shm.close()
    shm.unlink()

The worker process attaches to the shared memory by name, creates a NumPy array backed by the shared buffer, and modifies it. The parent sees the modification because both processes access the same physical memory. No data is copied or serialized.

shared_memory is the fastest way to share large data between processes. The tradeoff is that you manage the memory lifecycle manually: you must close() and unlink() the shared memory block when done, and you must coordinate access to avoid race conditions.

Comparing the approaches

The performance differences are significant. Accessing a Value takes nanoseconds. Accessing a managed list element takes microseconds because every operation involves serialization, IPC, and a round trip to the manager server.

For programs that exchange data frequently, choosing the wrong mechanism can dominate runtime.

The simplest correct approach is often the best starting point. Use Queue for message passing, Value for numeric counters, and Manager only when you need to share a complex Python object and the performance cost is acceptable for your workload. Move to shared_memory only when profiling shows that serialization is your bottleneck.