Synchronize Python Threads with Locks

When two Python threads read and write the same variable without coordination, the result is unpredictable. One thread can read a value, another can change it, and the first thread writes back a stale value, silently losing the update. This is a race condition, and it is the most common class of bug in multithreaded Python programs.

Python's threading.Lock prevents race conditions by ensuring that only one thread can access a protected section of code at a time. This article shows you how to use locks correctly, how to avoid the deadlocks and performance problems that misuse creates, and when to reach for RLock instead of Lock. If you have worked through creating and managing Python threads, you already know how to start and join threads; this article covers the synchronization primitives that make those threads safe to use together.

A lock is a simple concept: it is either held or not held. When a thread acquires a lock, no other thread can acquire it until the first thread releases it. If another thread tries to acquire a held lock, it blocks until the lock becomes available.

This serialization guarantees that the code between acquire and release runs in only one thread at a time. The challenge is using this simple tool in ways that are correct under all possible thread interleavings, not just the ones you tested.

What a race condition looks like

The simplest demonstration of a race condition is two threads incrementing a shared counter. Even though count += 1 looks like one operation, Python executes it as three separate bytecode instructions: read the value, add one, write it back. Two threads interleaving these steps produce incorrect results.

python

import threading
 
counter = 0
iterations = 1_000_000
 
def increment():
    global counter
    for _ in range(iterations):
        counter += 1
 
t1 = threading.Thread(target=increment)
t2 = threading.Thread(target=increment)
 
t1.start()
t2.start()
t1.join()
t2.join()
 
print(f"Expected: {iterations * 2}")
print(f"Actual:   {counter}")

Both threads run the same loop at the same time with no coordination, so their reads and writes to counter interleave unpredictably from one run to the next. One run on this machine might produce:

text

Expected: 2000000
Actual:   1456723

The actual value is less than expected because both threads sometimes read the same value, increment it, and write it back. One increment overwrites the other, and the lost updates accumulate over a million iterations. The exact shortfall varies unpredictably between runs because it depends on the timing of thread context switches.

This non-determinism is what makes race conditions hard to debug: a test that passes ten times can fail on the eleventh run.

Fixing the race condition with threading.Lock

A Lock serializes access to the critical section. Only one thread can hold the lock at a time, so the read-increment-write sequence becomes effectively atomic with respect to other threads.

python

import threading
 
counter = 0
lock = threading.Lock()
iterations = 1_000_000
 
def increment():
    global counter
    for _ in range(iterations):
        with lock:
            counter += 1
 
t1 = threading.Thread(target=increment)
t2 = threading.Thread(target=increment)
 
t1.start()
t2.start()
t1.join()
t2.join()
 
print(f"Expected: {iterations * 2}")
print(f"Actual:   {counter}")

Now every increment happens inside the with lock: block, so no thread can read the old value while another thread is in the middle of writing a new one. Running this version repeatedly always produces the correct total:

text

Expected: 2000000
Actual:   2000000

The with lock: statement acquires the lock before entering the block and releases it when the block exits, even if an exception occurs. This is the recommended pattern for lock usage in Python. The equivalent manual acquire and release is more verbose and error-prone because you must remember to call release() in a finally block.

The lock works, but notice that the locked version is much slower than the buggy version. Acquiring and releasing a lock two million times adds significant overhead. In practice, you should restructure the code to lock less frequently.

Instead of locking inside the loop, each thread can compute its contribution locally and only hold the lock while updating the shared counter once at the end.

python

def increment_batched():
    global counter
    local = 0
    for _ in range(iterations):
        local += 1
    with lock:
        counter += local

This version acquires the lock once per thread instead of once per iteration. The local accumulation runs without any lock contention, and the final update to the shared counter is brief. This pattern of moving work outside the critical section is essential for performant threaded code.

The context manager pattern for locks

Always use locks with the with statement. It is shorter, clearer, and guarantees the lock is released no matter how the block exits.

python

lock = threading.Lock()
 
# Recommended
with lock:
    shared_resource.update()
 
# Equivalent but error-prone
lock.acquire()
try:
    shared_resource.update()
finally:
    lock.release()

If an exception occurs between acquire() and try in the manual version, the lock is never released. If you forget the finally block, an exception leaves the lock held forever, and every other thread blocks indefinitely waiting for it. The with statement eliminates both risks.

Lock.acquire() accepts an optional blocking parameter and a timeout. acquire(blocking=False) returns True if the lock was acquired and False if it was already held, without blocking. acquire(timeout=2) blocks for at most 2 seconds and returns False if the lock could not be acquired in that time.

python

if lock.acquire(timeout=5):
    try:
        do_work()
    finally:
        lock.release()
else:
    print("Could not acquire lock within 5 seconds.")

The timeout pattern is useful for deadlock prevention and graceful degradation. If a thread cannot acquire a lock within a reasonable time, it can log a warning, retry later, or fall back to a degraded mode rather than blocking forever.

Protecting shared data structures

Locks are not just for integers. Custom objects and any code that combines a read with a write on shared state need protection, and this is the safer default to reach for once more than one step is involved.

python

import threading
 
shared_list = []
list_lock = threading.Lock()
 
def append_items(start):
    for i in range(start, start + 1000):
        with list_lock:
            shared_list.append(i)
 
threads = [
    threading.Thread(target=append_items, args=(0,)),
    threading.Thread(target=append_items, args=(1000,)),
    threading.Thread(target=append_items, args=(2000,)),
]
 
for t in threads:
    t.start()
for t in threads:
    t.join()
 
print(f"List length: {len(shared_list)}")
print(f"Sorted check: {sorted(shared_list) == list(range(3000))}")

All three threads append 1,000 numbers each, and the lock keeps their appends from interleaving badly enough to lose entries, so the final list always ends up with all 3,000 items in a valid, sortable state:

text

List length: 3000
Sorted check: True

A single call to list.append() is actually atomic in CPython and would not corrupt shared_list even without the lock here. The lock still matters in real code because most programs do more than a single append: they check a condition, update more than one shared value together, or read a value before writing it back, and that kind of multi-step sequence is not atomic even when each individual step is. Once your logic spans more than one operation on shared state, protect the whole sequence with a lock rather than relying on the safety of any single call.

Thread-safe alternatives exist for some data structures. queue.Queue is thread-safe by design and does not need external locking for put() and get(). collections.deque with maxlen supports thread-safe append() and popleft() in CPython, but this is an implementation detail, not a language guarantee.

When in doubt, use a lock.

Deadlocks and how to prevent them

A deadlock occurs when two or more threads each hold a lock the others need and none can proceed. The classic scenario involves two locks acquired in different orders by different threads.

python

import threading
 
lock_a = threading.Lock()
lock_b = threading.Lock()
 
def thread_1_work():
    with lock_a:
        print("Thread 1 acquired lock A")
        # Simulate some work
        import time; time.sleep(0.1)
        with lock_b:
            print("Thread 1 acquired lock B")
 
def thread_2_work():
    with lock_b:
        print("Thread 2 acquired lock B")
        import time; time.sleep(0.1)
        with lock_a:
            print("Thread 2 acquired lock A")

If thread 1 acquires lock_a and thread 2 simultaneously acquires lock_b, both threads will block forever waiting for the other lock. Thread 1 waits for lock_b, which thread 2 holds. Thread 2 waits for lock_a, which thread 1 holds.

Neither can make progress.

The fix is to always acquire multiple locks in a consistent global order. If every thread acquires lock_a before lock_b, deadlock is impossible because no thread can hold lock_b while waiting for lock_a.

python

def thread_2_work_fixed():
    with lock_a:          # Acquire in the same order as thread 1
        print("Thread 2 acquired lock A")
        import time; time.sleep(0.1)
        with lock_b:
            print("Thread 2 acquired lock B")

A second prevention strategy is to use timeouts. If a thread cannot acquire a lock within a deadline, it releases the locks it already holds and retries. This avoids permanent deadlock at the cost of introducing retry logic.

python

def acquire_locks_with_timeout():
    while True:
        lock_a.acquire()
        if lock_b.acquire(timeout=1):
            break
        lock_a.release()
        import time; time.sleep(0.1)
    # ... work with both locks ...
    lock_b.release()
    lock_a.release()

This pattern, called lock ordering with backoff, is useful in complex systems where a strict global lock order is difficult to enforce statically. For most programs, consistent lock ordering is simpler and sufficient.

RLock: the reentrant lock

A standard Lock cannot be acquired twice by the same thread. If a thread that already holds a lock tries to acquire it again, it deadlocks against itself. RLock (reentrant lock) allows the same thread to acquire the lock multiple times.

The lock is released only when the thread calls release() the same number of times it called acquire().

python

import threading
 
rlock = threading.RLock()
 
def outer():
    with rlock:
        print("Outer acquired lock")
        inner()
 
def inner():
    with rlock:
        print("Inner acquired lock")
 
threading.Thread(target=outer).start()

Both outer() and inner() acquire the same rlock from the same thread, and because RLock allows that, execution reaches both print statements instead of hanging on the second acquisition:

text

Outer acquired lock
Inner acquired lock

With a standard Lock, the call to inner() would block forever because outer() already holds the lock and inner() tries to acquire it again. RLock tracks which thread holds it and a recursion count, so reentrant acquisition succeeds.

Use RLock when you have functions that call each other and need the same lock, like a class where a public method acquires a lock and calls a private helper that also needs the lock:

python

class ThreadSafeCache:
    def __init__(self):
        self._lock = threading.RLock()
        self._data = {}
 
    def get(self, key):
        with self._lock:
            return self._data.get(key)
 
    def get_or_compute(self, key, factory):
        with self._lock:
            if key not in self._data:
                self._data[key] = factory()  # May call self.get()
            return self._data[key]

If factory() calls self.get() on the same cache, an RLock handles it correctly. A standard Lock would deadlock. Use RLock only when reentrancy is a real need.

Standard Lock is simpler, slightly faster, and sufficient for most synchronization.

Lock granularity and performance

How much code you place inside a locked section directly affects your program's performance. A lock that guards a large block of code serializes threads and reduces concurrency. A lock that guards too little code risks data races.

Finding the right granularity is a balance.

Holding a lock during a slow I/O call is a common mistake, because every other thread that needs the same lock has to wait out the full duration of that call even though the I/O itself does not touch shared state:

python

with lock:
    data = fetch_from_api()       # Takes 2 seconds, other threads wait
    results.append(data)

Moving the I/O call outside the with block fixes this. The lock is only held long enough to append the already-fetched data, so other threads can keep fetching their own data while this thread updates the shared list:

python

data = fetch_from_api()           # No lock, other threads can also fetch
with lock:
    results.append(data)

The same problem shows up in loops. Acquiring and releasing the lock on every iteration adds overhead on every single item and serializes work that does not need to be serialized:

python

for item in items:
    with lock:
        counter += process(item)

Accumulating into a local variable first and locking only once at the end keeps the critical section small no matter how many items the loop processes:

python

local_count = 0
for item in items:
    local_count += process(item)
with lock:
    counter += local_count

The principle is to hold locks for the minimum time necessary to safely access shared state. Compute locally, I/O without the lock, and only synchronize at the point where shared state is read or written.

Using threading events for signaling

Locks protect shared data. Events coordinate control flow between threads. An Event is a simple boolean flag that one thread sets and other threads wait for.

python

import threading
import time
 
data_ready = threading.Event()
shared_data = None
 
def producer():
    global shared_data
    time.sleep(2)
    shared_data = {"status": "ready"}
    data_ready.set()
 
def consumer():
    print("Consumer waiting for data...")
    data_ready.wait()
    print(f"Consumer got: {shared_data}")
 
t1 = threading.Thread(target=producer)
t2 = threading.Thread(target=consumer)
 
t2.start()
t1.start()
t1.join()
t2.join()

The consumer thread starts first and immediately blocks on data_ready.wait(), so it prints its waiting message right away and only prints the data once the producer thread sets the event two seconds later:

text

Consumer waiting for data...
(2 second pause)
Consumer got: {'status': 'ready'}

Event.wait() blocks until Event.set() is called. You can pass a timeout to wait() to avoid blocking forever. Event.clear() resets the flag to False, and Event.is_set() checks the current state without blocking.

Events are simpler than locks for one-shot signaling. Use an event when one thread produces a result that other threads consume after it is ready. Use a lock when multiple threads read and write shared state over time.

Combining locks with higher-level primitives

The threading module includes synchronization primitives that combine locks with additional semantics:

• Condition wraps a lock and adds wait(), notify(), and notify_all(). Use it when threads need to wait for a specific state change, not just for a lock to become available. A common pattern is a producer-consumer setup where consumers wait until items are available, which thread communication with queues covers using the higher-level queue.Queue API.

• Semaphore is a counter-based lock. A semaphore initialized to N allows up to N threads to acquire it concurrently. Use it to limit concurrent access to a resource like a database connection pool or a rate-limited API.

• Barrier synchronizes a fixed number of threads at a rendezvous point. Every thread calls barrier.wait() and blocks until the required number of threads have arrived. Use it when a group of threads must all complete a phase before any can start the next phase.

These primitives are built on top of Lock and use the same acquire and release semantics internally. Understanding locks gives you the foundation to use all of them correctly. If your coordination needs go beyond simple mutual exclusion, the next articles in this section cover these primitives in detail.

Locking checklist before you write threaded code

Before adding locks to a program, verify that you actually need them. If each thread works on its own data and never shares state with other threads, locks are unnecessary. If threads only communicate through queue.Queue, which is internally synchronized, you still do not need locks.

When you do need locks, follow these rules:

• Identify every mutable object that multiple threads access.
• Determine which accesses are reads and which are writes. If all accesses are reads, no lock is needed.
• Group related updates into a single critical section protected by one lock.
• Keep locked sections short. Move computation and I/O outside the lock.
• Always use with lock: rather than manual acquire and release.
• If multiple locks are needed, establish a consistent acquisition order.
• Test with more threads than CPU cores to increase the chance of exposing race conditions.