Optimize Python Loops and Iterations

Python loops are often where a program spends most of its time. A loop that runs a million times magnifies small inefficiencies into noticeable slowdowns, so learning to optimize Python loops is high-leverage work because improvements apply to every iteration.

The techniques here assume you have already used a profiler to identify which loops are slow. If you have not profiled yet, start with the article on profiling Python bottlenecks with cProfile to find the loops worth optimizing.

Why Python loops are slower than you expect

Every iteration of a Python loop pays interpreter overhead. The interpreter evaluates the loop condition, retrieves the next element from the iterator, updates internal state, and dispatches the loop body. None of this overhead exists in compiled code.

python

total = 0
for i in range(1_000_000):
    total += i

Each iteration involves creating an integer object for i, looking up the total variable, calling __iadd__, and assigning the result back. A million iterations of a loop body that is just integer addition can take tens of milliseconds. The same loop in C would be microseconds.

The goal of loop optimization is not to eliminate the loop entirely. It is to reduce the per-iteration cost by moving work out of the loop body and into faster execution paths, ideally into C-level built-in functions or comprehensions.

Hoist invariants out of the loop

A loop invariant is a value or computation that produces the same result every iteration. Computing it inside the loop wastes time.

python

import math
 
# Before: len(data) and math.sqrt called every iteration
def compute(data, factor):
    results = []
    for item in data:
        if len(data) > 10:
            results.append(item * math.sqrt(factor))
    return results
 
# After: invariants hoisted before the loop
def compute(data, factor):
    results = []
    n = len(data)
    sqrt_factor = math.sqrt(factor)
    if n > 10:
        for item in data:
            results.append(item * sqrt_factor)
    return results

The optimized version calls len(data) and math.sqrt(factor) exactly once each, then reuses the results. If data has a million elements, len(data) and math.sqrt(factor) inside the loop would execute a million times each.

More subtly, the condition len(data) > 10 does not depend on the loop variable. Checking it once before the loop avoids checking it on every iteration. The structure changes from "check, then maybe append" to "check once, then loop unconditionally."

Cache attribute lookups in local variables

Every dotted name lookup in Python involves a dictionary search. obj.attr looks up attr in type(obj).__dict__ and possibly in parent types. Inside a loop, this lookup repeats on every iteration.

python

# Before: data.append resolved every iteration
def build_list(data):
    result = []
    for item in data:
        result.append(item)
    return result
 
# After: append cached as a local variable
def build_list(data):
    result = []
    append = result.append
    for item in data:
        append(item)
    return result

The speed difference is measurable. append = result.append captures the bound method once. Inside the loop, append(item) is a local variable lookup, which is a C-level array index rather than a dictionary search.

This technique applies to any dotted lookup inside a hot loop: module-level functions, object methods, and chained attribute access.

python

# Before
for item in data:
    math.sqrt(item)
    config.get("key")
    obj.method()
 
# After
sqrt = math.sqrt
get = config.get
method = obj.method
for item in data:
    sqrt(item)
    get("key")
    method()

Use list comprehensions instead of manual for-loops

A list comprehension executes its loop in C rather than Python bytecode. For simple transformations, the speed difference is 20 to 50 percent.

python

# Manual for-loop
squares = []
for i in range(1000):
    squares.append(i * i)
 
# List comprehension
squares = [i * i for i in range(1000)]

The comprehension version avoids the per-iteration overhead of calling append explicitly and checking the loop condition in bytecode. The entire loop body runs in the C evaluation loop.

Comprehensions also support filtering, which replaces a manual loop with a conditional append:

python

# Manual for-loop with filter
evens = []
for i in range(1000):
    if i % 2 == 0:
        evens.append(i)
 
# Comprehension with filter
evens = [i for i in range(1000) if i % 2 == 0]

The same speedup applies beyond lists. Set and dictionary comprehensions provide the same benefit for their respective types, running the loop and the membership or key insertion in C instead of Python bytecode:

python

unique = {i % 10 for i in range(1000)}
mapping = {i: i * i for i in range(1000)}

The comprehension advantage is largest when the loop body is a single expression. For multi-statement loop bodies, a manual loop is necessary, but you can still apply the other techniques here.

Use built-in functions for common patterns

Python's built-in functions are implemented in C. Replacing a manual loop with a built-in function moves the iteration from Python bytecode into C.

python

# Manual loop
total = 0
for num in data:
    total += num
 
# Built-in sum
total = sum(data)

The sum() version runs the accumulation loop entirely in C. The speedup is typically 3x to 10x for large datasets.

Other high-leverage built-ins for replacing loops:

python

any(data)         # instead of loop with or
all(data)         # instead of loop with and
max(data)         # instead of tracking maximum manually
min(data)         # instead of tracking minimum manually
sorted(data)      # returns a new sorted list
enumerate(data)   # instead of range(len(data))
zip(a, b)         # instead of parallel indexing

The map() and filter() built-ins are also C-level loops, but they return iterators rather than concrete collections. Use them when a comprehension would do extra work:

python

# map with a built-in function runs in C
strings = list(map(str, range(1000)))
 
# filter with a built-in function
positives = list(filter(None, data))

When the mapping function is a built-in like str, int, or float, map() is fast because the function call also happens in C. When the function is a lambda, map() offers little performance advantage over a comprehension and is usually less readable.

Choose the right data structure for lookups inside loops

A loop that performs a membership test or a lookup on every iteration is dominated by the cost of that operation. Using the wrong data structure turns an O(n) loop into O(n²).

python

# O(n²): list membership inside a loop
def find_common(list_a, list_b):
    common = []
    for item in list_a:
        if item in list_b:
            common.append(item)
    return common
 
# O(n): set membership inside a loop
def find_common(list_a, list_b):
    set_b = set(list_b)
    common = []
    for item in list_a:
        if item in set_b:
            common.append(item)
    return common

The set version converts list_b to a set once, then performs O(1) membership tests inside the loop. With a thousand items in each list, the first version does up to a million comparisons. The second version does about a thousand.

This pattern applies to any loop that looks up values by key. If the loop body does data[key] or key in data and data is a list or tuple, consider whether a dictionary or set would be the right container. The article on Python collections covers how each container behaves.

Avoid re-evaluating conditions inside loops

When a loop body contains an if statement where the condition is the same for every iteration, pull the condition outside the loop and duplicate the loop.

python

# Before: condition checked on every iteration
def process(items, mode):
    result = []
    for item in items:
        if mode == "fast":
            result.append(item * 2)
        else:
            result.append(item + 1)
    return result
 
# After: condition evaluated once
def process(items, mode):
    result = []
    if mode == "fast":
        for item in items:
            result.append(item * 2)
    else:
        for item in items:
            result.append(item + 1)
    return result

The second version duplicates the loop but evaluates mode == "fast" exactly once. For a million-iteration loop, this saves a million comparisons. The code is longer but measurably faster.

This technique is worth applying when the loop body is simple relative to the condition check and the iteration count is high. For loops with complex bodies or low iteration counts, the code duplication is not worth the marginal speedup.

Pre-allocate lists when the size is known

list.append() occasionally triggers a resize when the internal array fills up. The resize allocates a larger array and copies all existing elements. For large lists, multiple resizes add noticeable overhead.

python

# Before: list grows dynamically
result = []
for i in range(1_000_000):
    result.append(i)
 
# After: pre-allocated with known size
result = [None] * 1_000_000
for i in range(1_000_000):
    result[i] = i

The pre-allocated version creates the list once at the final size, then assigns to each index. No resizes occur. The tradeoff is that the list comprehension version is both faster and more readable for this specific case:

python

result = [i for i in range(1_000_000)]

Pre-allocation is most useful when the loop body is complex and cannot be expressed as a simple comprehension. In those cases, pre-allocating avoids the resize overhead without sacrificing readability.

Minimize function calls inside loops

A function call in Python has overhead: the interpreter pushes a new frame, resolves arguments, and transfers control. Inside a hot loop, even cheap function calls can add up.

python

# Before: helper function called per iteration
def is_valid(item):
    return item > 0 and item % 2 == 0
 
result = [item for item in data if is_valid(item)]
 
# After: condition inlined
result = [item for item in data if item > 0 and item % 2 == 0]

Inlining the condition removes a million function calls if data has a million elements. The inline version is sometimes less readable, so apply this technique only when the function is simple and the performance gain is confirmed by measurement.

For functions that must remain separate for correctness or clarity, caching the function reference as a local variable still helps:

python

is_valid_local = is_valid
result = [item for item in data if is_valid_local(item)]

Know when to stop optimizing

Not every loop needs optimization. A loop that runs ten times with a 10 ms body is not a bottleneck. A loop that runs a million times with a 1 µs body might be.

Apply these techniques in order of effort:

1. Hoist invariants and cache lookups: low effort, always worth doing in hot loops.

2. Replace manual loops with comprehensions or built-ins: low effort when the logic is simple.

3. Choose the right data structure: medium effort, high payoff when the current structure is wrong.

4. Pre-allocate or restructure conditions: apply only when measurements show the loop is still too slow.

Measure before and after every change with timeit. A change that feels faster may not actually be faster. A change that makes the code harder to read without measurable improvement is a net loss.

The broader topic of Python performance optimization covers the full workflow from profiling through optimization. When the bottleneck shifts from loop speed to memory usage, the article on writing memory-efficient Python code covers techniques like generators, __slots__, and streaming.