Common Python Iterator and Generator Mistakes

Every Python feature that is powerful enough to change how you write code also comes with a set of mistakes that are common enough to have names. Iterators and generators are no exception. The very qualities that make them valuable, lazy evaluation, one-shot consumption, and the ability to pause and resume, are also the source of the bugs that trip up developers who are new to the iteration model. The good news is that these mistakes follow predictable patterns, and once you have seen each one a few times, you learn to spot them before they cause problems.

This article collects the mistakes that appear most frequently in real Python codebases, from beginner scripts to production systems. Each one is accompanied by an explanation of why it happens and a clear fix. If you have worked through the earlier articles on Python generators, generator expressions, and passing values with yield, you already have the conceptual foundation to understand why these mistakes occur and how the fixes work.

Exhaustion: trying to iterate twice

The single most common generator mistake is assuming you can iterate over a generator more than once. A generator is an iterator, and iterators are consumed as you use them. After a for loop has walked through every value a generator can produce, the generator is empty forever. A second for loop over the same generator object produces no output:

python

gen = (x * 2 for x in range(3))
print(list(gen))
print(list(gen))

The first list() call returns [0, 2, 4]. The second returns an empty list. The generator did not reset, and it does not store its values for replay. This behaviour is by design. Generators are intended for one-pass data processing where values flow through once and are gone. If you need to iterate multiple times, you have two options. The first is to call the generator function or recreate the generator expression each time you need a new iterator. The second is to materialize the values into a list or tuple if the dataset is small enough to fit in memory.

The exhaustion mistake often appears in subtler forms. A function that accepts an iterable argument might consume it with a for loop, not realizing that the caller intended to iterate over the same data again later. If the caller passes a generator, the function consumes it, and the caller's subsequent iteration finds nothing. The fix is either to document that the function consumes its iterable argument, or for the function to materialize the iterable into a list internally if it needs multiple passes.

Late binding in generator expressions

Generator expressions capture variables from their enclosing scope by name, not by value. If the captured variable changes between the time the generator expression is created and the time it is consumed, the generator sees the changed value. This is the same late-binding behaviour that affects closures and lambda expressions in Python:

python

funcs = []
for i in range(3):
    funcs.append(lambda: i)
print([f() for f in funcs])

All three lambdas print 2, because i is captured by name and its final value after the loop is 2. Generator expressions have the same behaviour. A generator expression created inside a loop that references the loop variable will see the loop variable's final value when consumed later:

python

generators = [(x for x in range(3) if x == i) for i in range(3)]
for g in generators:
    print(list(g))

You might expect three generators, each filtering for a different value of i. In practice, all three generators use the final value of i, which is 2, because the generator expression captures i by name and i changes with each iteration of the outer list comprehension. The fix is to consume the generator expression immediately while the variable still holds the intended value, or to bind the value in a way that survives the loop, such as using a default argument.

Accidental materialization

The whole point of generators is to avoid building lists. But it is surprisingly easy to accidentally materialize a generator into a list without realizing it. Functions like sorted(), list(), and tuple() consume an iterator and build a complete collection in memory. If you pass a generator that produces ten million values to sorted(), Python materializes all ten million values into a list before it can begin sorting, which defeats the memory-efficiency purpose of the generator.

A more subtle form of accidental materialization happens when you store generator output in a data structure. Appending generator values to a list one at a time in a loop eventually builds a list of all the values, which is effectively the same as calling list() on the generator. If the intent was to process values and discard them, the loop should process each value and move on without storing it:

python

def large_generator():
    return (x * x for x in range(10_000_000))
 
results = []
for value in large_generator():
    results.append(value + 1)

This code builds a list of processed values that grows with the size of the generator's output. If the generator produces a million values, results will be a million-element list. The memory-efficient version processes each value and writes it out or aggregates it without storing it:

python

total = 0
for value in large_generator():
    total += value + 1

The difference is the data structure. The first version accumulates. The second version aggregates. Knowing which one you need, accumulation or aggregation, is the key to avoiding accidental materialization.

Misusing send() on generators

The send() method on generators has a specific protocol that must be followed exactly. A freshly created generator has not reached a yield expression, so calling send() with a non-None value raises TypeError. The generator must be primed with next() or send(None) before any meaningful value can be sent. Forgetting to prime the generator is the most common send()-related mistake.

A related mistake is assuming that send() and next() are interchangeable in a bidirectional generator. They are not. next() implicitly passes None into the generator as the result of the yield expression. If your generator uses the received value for calculations, passing None when you meant to pass a number will cause a TypeError when the generator tries to add None to an integer. The caller must track whether each step should use next() or send(), and must always pass a value of the type the generator expects.

The cleanest way to avoid send() mistakes is to avoid send() unless you genuinely need bidirectional communication. Most generator use cases are one-way: the generator produces values, and the caller consumes them. For one-way generators, next() and for loops are all you need. Reserve send() for the specific patterns where the generator acts as a coroutine that processes incoming data, such as a running average calculator or a state machine that transitions based on external input.

Expecting random access or length from generators

Generators do not support indexing. You cannot write gen[5] to get the sixth value from a generator. Generators also do not support len(). You cannot ask a generator how many values it will produce without consuming all of them. These limitations are not bugs; they are consequences of lazy evaluation. The values do not exist until they are computed, so there is nothing to index and nothing to count.

The mistake is using a generator where the algorithm requires random access. If you need to look up the hundredth item, or compare the first and last items, or check how many items there are before processing, you need a sequence, not a generator. Convert the generator to a list if the data fits in memory, or redesign the algorithm to use sequential access if it does not. The choice between a generator and a list is not about style; it is about whether your algorithm needs random access or only sequential access.

A subtler version of this mistake is using a generator with a function that internally requires random access. The random.shuffle() function shuffles a sequence in place, but it cannot shuffle a generator because there is no sequence to rearrange. The max() and min() functions work fine with generators because they only need sequential access to find the extreme values. Before passing a generator to any function, ask yourself whether that function needs to know the length of its input, index into it, or iterate over it multiple times. If the answer is yes, the function needs a sequence, not a generator.