Python Iterators vs Generators

After reading the articles in this section on Python iterables and iterators, creating custom iterators, and Python generators, you have two different tools that solve the same fundamental problem: producing a sequence of values one at a time. Both implement the iterator protocol. Both work with for loops, list comprehensions, and every built-in function that accepts iterables. Both support lazy evaluation. The question is not which one is better in absolute terms, but which one is better for the specific piece of code you are writing right now.

The relationship between iterators and generators is straightforward: every generator is an iterator, but not every iterator is a generator. A generator is a specific kind of iterator created by a generator function using yield or by a generator expression. When you write a generator function, Python automatically creates an object that implements iter() and next() for you. You never write those methods yourself. A custom iterator, by contrast, is a class where you explicitly implement iter() returning self and next() returning the next item or raising StopIteration. Both approaches produce objects that are indistinguishable to the code that consumes them, but they represent fundamentally different authoring experiences.

Choosing between them is a design decision that affects how you write, test, debug, and maintain your code. The decision is rarely permanent. You can start with a generator for rapid prototyping, discover that you need more control, and refactor to a custom iterator class later without changing any of the consuming code. The consuming code only sees an iterator, so the implementation behind that interface can evolve freely.

The case for generators

Generators are the default choice for most value-production tasks, and for good reason. A generator function strips away every line of code that is not directly about producing values. There is no class declaration, no init to store state, no iter returning self, and no next with an explicit StopIteration check. You write the logic of what values to produce, and Python handles the mechanics of how to produce them.

Consider the difference in line count. A generator that yields Fibonacci numbers takes six lines of actual logic. The equivalent custom iterator class takes roughly twenty lines, most of which are boilerplate that repeats across every iterator you write. The generator version is not just shorter; it is clearer, because the boilerplate is absent and the value-production logic is the only thing on the screen.

Generators also eliminate an entire category of bugs related to the iterator protocol. In a custom iterator class, you must remember to raise StopIteration at the right time, return self from iter(), and manage your position state correctly across calls to next(). Forgetting any of these produces an iterator that silently produces wrong results, runs forever, or raises exceptions at surprising times. Generator functions handle all of this automatically. The function body ending naturally raises StopIteration. The generator object's iter() returns self. The local variables are preserved correctly between calls without you managing any state manually.

For one-shot iteration where the values are produced once and never revisited, generators are almost always the right answer. Reading lines from a file, consuming messages from a queue, fetching pages from an API, processing sensor readings, and transforming data in a pipeline are all tasks where the generator's one-pass nature matches the problem perfectly. The data source itself is one-pass, so the iterator being one-pass is a feature, not a limitation.

The case for custom iterators

Custom iterator classes earn their place when the generator model is too simple for the problem. The most common reason to write a custom iterator is reusability. A generator object can only be iterated once. If you need to iterate over the same logical sequence multiple times, you must either recreate the generator each time by calling the generator function again, or you must separate the data source from the traversal state using the two-class pattern. The two-class pattern, where an iterable class stores data and creates fresh iterator instances, requires at least one class, and at that point you have already committed to the class-based approach.

Another strong reason for custom iterators is when the iterator needs a public API beyond next(). A generator object exposes send(), throw(), and close() in addition to the standard iterator protocol, but you cannot add custom methods to it. If your iterator needs a reset() method to restart from the beginning, a progress() method to report how far through the data it is, or a skip_to() method to jump ahead, you need a class. The class's methods can inspect and modify the internal state in ways that a generator's opaque local variables cannot support.

Complex state is the third major reason to prefer a class. A generator's state is distributed across its local variables and its current execution point. For simple state like a counter and a limit, this is perfectly adequate. For complex state like a tree traversal stack, a tokenizer buffer with lookahead, or a state machine with multiple interacting variables, the class model with named attributes is easier to reason about and debug. You can inspect the state at any point by looking at the attributes, and you can write tests that create an iterator, advance it partway, and assert on specific attribute values, something that is awkward to do with a generator's opaque internal state.

How to decide in practice

When you sit down to write code that produces a sequence of values, ask yourself three questions. First, does the consumer need to iterate over this sequence more than once? If yes, you need either a generator function that can be called multiple times, each call producing a fresh generator, or a custom iterable class that creates fresh iterators on demand. Both approaches work; the class approach is more explicit about the reusability guarantee.

Second, does the iterator need methods beyond the standard iterator protocol? If you find yourself wanting a method that inspects or modifies the iterator's state, a class is the right choice. Generator objects do not support custom methods, and trying to work around this by passing state through send() usually results in code that is harder to follow than the equivalent class.

Third, how complex is the state that drives the value production? If the state is a single counter, a generator is perfect. If the state is a stack, a queue, a buffer, and a flag, with branching logic that depends on combinations of these, the class model with named attributes will be easier to write correctly and easier to debug when something goes wrong.

When the answers to these questions are mixed, you have to weigh the tradeoffs. A generator with slightly complex state is still shorter than a class with simple state. A class with a single extra method is still more boilerplate than a generator without one. Experience teaches you where the threshold lies, but the good news is that the consuming code does not care which you choose. You can start with a generator, discover that you need a progress() method, and refactor to a class without changing a single line of the code that iterates over the values.

Memory and performance considerations

Generators and custom iterators have essentially identical memory and performance characteristics because both implement the same lazy-evaluation model. A generator that yields integers one at a time uses the same amount of memory as a custom iterator that computes integers one at a time, roughly the size of the current integer plus the object overhead. Neither stores a list of values, and neither computes values before they are requested.

The performance difference between generators and custom iterators is negligible for most applications. Generators have slightly more function-call overhead because each yield involves saving and restoring the Python frame, while a custom iterator's next() is a plain method call. For iterators that produce millions of values per second, this overhead can become measurable. But for the vast majority of Python programs, where iteration speed is dominated by the work done inside the loop body, the difference between a generator and a custom iterator is lost in the noise.

The more important performance consideration is whether you are accidentally materializing a generator into a list when a generator would suffice. Code that calls list() on a generator and then iterates over the list has lost the memory benefit of the generator. Code that passes a generator to a function that internally materializes it, like sorted() which must build a list to sort it, is using memory proportional to the input size regardless of whether the input is a generator or a list. Understanding what each consumer does with its input is more important than choosing between a generator and a custom iterator.

Combining both approaches

You are not forced to choose one approach for an entire codebase. The most effective Python code uses generators for simple, linear value production and custom iterators for complex or reusable sequences. A library might expose custom iterator classes in its public API, giving users objects with documented methods and predictable behaviour, while using generators internally for data transformation pipelines. The consuming code does not know or care which approach produced the iterator it is looping over.

Generators are also useful inside custom iterator classes. A class's iter() method can be a generator function, which gives you the reusability of the two-class pattern without writing a separate iterator class. The iter() method yields values, and each call creates a fresh generator object. This pattern combines the simplicity of generators with the reusability of classes and is often the best of both worlds for moderately complex iterables.

The iterator protocol's design, where any object with iter() and next() works with for loops and built-in functions, is the reason this flexibility exists. Python does not care whether the iterator was created by a generator function, a generator expression, or a hand-written class. It only cares that the object responds correctly to iter() and next(). This uniform interface is one of Python's greatest design strengths, and it means you can choose the implementation approach that best fits your problem without worrying about compatibility.