Common Python Performance Bottlenecks

Most Python performance bottlenecks fall into a handful of patterns. Recognizing them saves hours of profiling. The same bottlenecks appear in web servers, data pipelines, and scripts alike.

Here are the most common Python bottlenecks, how to spot them, and how to fix each one.

Bottleneck 1: I/O without concurrency

Your program spends 80 percent of its time waiting for network responses, disk reads, or database queries. The CPU sits idle while the I/O completes.

The fix is not to make the I/O faster. It is to overlap waits with other work. Threading or asyncio lets you start the next request while the first one is still waiting.

python

# Sequential: total time = sum of all waits
results = []
for url in urls:
    data = fetch(url)
    results.append(process(data))
 
# Concurrent: total time = max single wait
from concurrent.futures import ThreadPoolExecutor
 
def fetch_and_process(url):
    return process(fetch(url))
 
with ThreadPoolExecutor(max_workers=10) as pool:
    results = list(pool.map(fetch_and_process, urls))

The first version waits for each URL before starting the next. The second fires all requests immediately. Total time drops from 10 seconds to 1 second for 10 URLs that each take 1 second.

Bottleneck 2: wrong data structure inside a loop

A membership test inside a loop turns an O(n) operation into O(n squared). You see it when a program that handles 1,000 items is fine but 100,000 items takes forever.

python

# O(n²): list membership inside loop
common = []
for item in items_a:
    if item in items_b:          # scans list_b every time
        common.append(item)
 
# O(n): convert to set first
set_b = set(items_b)
common = [item for item in items_a if item in set_b]

The list version scans items_b from start to finish for every element of items_a. With 10,000 items each, that is 100 million comparisons. The set version does 10,000 hash lookups, each O(1).

The same pattern applies to dictionary lookups (use a dict, not a list of tuples) and counting frequencies (use collections.Counter, not a manual loop).

Bottleneck 3: repeated attribute lookups

Every dot in obj.method or module.function triggers a dictionary lookup. Inside a hot loop, this overhead adds up.

python

# Before: result.append resolved every iteration
result = []
for item in data:
    result.append(item)
 
# After: cache in local variable
result = []
append = result.append
for item in data:
    append(item)

The difference is measurable. append = result.append captures the bound method once. The loop body uses a C-level local variable lookup instead of a dictionary search.

For module functions, import them directly into a local name:

python

from math import sqrt

Bottleneck 4: repeated computation

A value that does not change inside a loop should not be computed inside the loop.

python

# Before: len(data) computed every iteration
for item in data:
    if len(data) > 100:
        process(item)
 
# After: compute once
n = len(data)
if n > 100:
    for item in data:
        process(item)

Hoisting the condition outside the loop also avoids checking it on every iteration. The loop becomes tighter and faster.

Bottleneck 5: materializing unnecessary collections

List comprehensions are great, but chaining them creates intermediate lists that are immediately discarded. Generator expressions avoid this.

python

# Creates three lists in memory
nums = [i for i in range(1_000_000)]
squares = [x * x for x in nums]
evens = [s for s in squares if s % 2 == 0]
result = sum(evens)
 
# No intermediate lists
result = sum(
    s for s in (x * x for x in range(1_000_000))
    if s % 2 == 0
)

The second version computes each value on demand and adds it to the sum. Memory usage is constant.

Bottleneck 6: GIL contention in CPU-bound threads

The Global Interpreter Lock prevents multiple threads from running Python bytecode simultaneously. For I/O-bound work, this is fine because threads release the GIL during I/O. For CPU-bound work, threading can actually make things slower.

The fix for CPU-bound work is multiprocessing, which spawns separate Python interpreters each with their own GIL. Or move the heavy computation into a C extension like NumPy, which releases the GIL during computation.

Bottleneck 7: parsing the same data repeatedly

Parsing JSON, XML, or CSV on every access is wasteful when the data does not change. Cache the parsed result.

python

@lru_cache(maxsize=1)
def get_config(path):
    with open(path) as f:
        return json.load(f)

The first call parses the file. Every subsequent call returns the cached dictionary. The article on Python caching covers caching in depth.

The root cause pattern

In practice, the same root causes appear across all bottlenecks: doing work in Python that could happen in C, repeating the same work unnecessarily, and waiting synchronously for I/O. When you spot one of these patterns, the fix is usually a built-in function, a better data structure, or concurrency.

Profiling shows you how to find these patterns in your own code. For data structure choices specifically, the article on optimizing Python collections covers the performance tradeoffs of each container type.