Python Iterator Protocol Explained

The Python iterator protocol is the formal contract that every Python iterator must follow. It is a simple protocol consisting of exactly two methods, iter() and next(), but its implications reach into every corner of the language. Every for loop you write, every comprehension you construct, and every call to built-in functions that consume iterables ultimately depends on objects that implement these two methods correctly. Understanding the protocol at this level is not an academic exercise; it is the bridge between using Python's built-in iteration tools and building your own iterable objects that integrate seamlessly with the language.

The protocol was designed during Python's early development to replace an older, less flexible model of iteration. Before Python 2.2, iteration was based solely on the getitem() method with sequential integer indexes. That approach worked for sequences like lists and tuples, but it could not support objects that generate values on the fly without storing them, like file readers or infinite streams. The iterator protocol, formalized in PEP 234, solved this by separating the concept of a data source from the concept of a traversal cursor. Any object can be iterable if it provides an iterator, and any object can be an iterator if it tracks position and yields items until exhaustion.

If you have read the introduction to Python iterables and iterators, you already know that iterables are objects that can produce iterators and iterators are objects that track position while producing items. The iterator protocol is the specific mechanism that makes this separation work. Let us examine each method in detail and see how Python's runtime uses them to power the for loop.

The iter() method

The iter() method has the simpler contract of the two. When called on an iterable, it must return an iterator object. For the iterable itself, this method creates and returns a new iterator that starts from the first element. For an iterator object, iter() must return self. This rule, that an iterator's iter() returns itself, is what makes every iterator also an iterable. It is the reason you can pass an iterator directly to a for loop without wrapping it in a call to iter().

This design choice, sometimes called the iterator self-iteration convention, simplifies code that works with iterables. Functions that accept iterables can begin by calling iter() on their argument without worrying about whether the argument is already an iterator. If it is, iter() returns it unchanged. If it is a list, tuple, or other non-iterator iterable, iter() creates a fresh iterator. The function then calls next() on whatever iter() returned, and the iteration proceeds correctly in both cases. This uniformity is what allows Python's for loop, the list() constructor, the tuple() constructor, and built-in functions like sum() and max() to accept any iterable without type-checking.

The iter() method should be fast and should not produce visible side effects. Its job is to set up the traversal, not to begin it. The actual work of producing values happens in next(). Separating setup from value production keeps the protocol clean and makes it possible to create multiple independent iterators from the same iterable, each managing its own traversal state without interference.

The next() method

The next() method does the real work of the iterator. Each call must return the next item in the sequence. When there are no more items, the method must raise the StopIteration exception. There is no other valid way to signal exhaustion. Returning a sentinel value like None or -1 is not part of the protocol and will cause for loops and other iteration consumers to continue indefinitely, which usually results in an infinite loop or an eventual crash.

The next() method is called implicitly by the for loop machinery and explicitly by the next() built-in function. In either case, the caller expects one of two outcomes: a valid return value representing the next item, or a StopIteration exception signalling that the sequence is finished. The iterator must maintain enough internal state between calls to know which item to produce next. This state is typically stored in instance attributes set up during init() and advanced during next().

Consider a minimal iterator that produces the numbers from 0 up to a given limit:

python

class CountUp:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0
 
    def __iter__(self):
        return self
 
    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

The class stores two pieces of state: the limit that defines when to stop, and the current position. Each call to next() checks whether the current position has reached the limit. If it has, the method raises StopIteration. If it has not, it records the current value, advances the position, and returns the recorded value. The iter() method returns self because each CountUp instance is its own iterator, consumed once and then exhausted.

You can use this iterator in a for loop directly because it implements both methods:

python

for num in CountUp(5):
    print(num)

The loop prints the numbers 0 through 4 and then stops, exactly as a range(5) loop would. The for loop calls iter() on the CountUp instance, gets self back, then calls next() repeatedly until StopIteration tells it to exit.

How the for loop uses both methods

Understanding the exact sequence of calls helps you debug iteration problems and design your own iterators correctly. When Python encounters a for loop, it executes these steps in order. First, it evaluates the expression after the in keyword to obtain the iterable object. Second, it calls iter() on that object, which in turn calls the object's iter() method, to obtain an iterator. Third, it enters a loop where each iteration calls next() on the iterator, which calls next(). The return value of next() is assigned to the loop variable, and the loop body executes. When next() raises StopIteration, the for loop catches it, breaks out of the loop, and continues execution after the loop body. If the loop body itself raises an exception, the for loop still performs cleanup on the iterator before propagating the exception.

This sequence is not just an implementation detail. It is part of Python's language specification, and every compliant Python interpreter must follow it. Jython, IronPython, PyPy, and MicroPython all implement the same protocol. Code that relies on the iterator protocol is therefore portable across all Python implementations, which makes it a stable foundation for library code that needs to work everywhere.

One subtle consequence of this protocol is that the for loop always calls iter() on its target, even if the target is already an iterator. For iterators that return self from iter(), this is a harmless no-op. But it means you cannot intercept or customize the iteration behaviour of an iterator at the for-loop level. The customization point is in the iterable that creates the iterator, not in the iterator itself.

The StopIteration contract

StopIteration is a built-in exception class whose sole purpose is to signal the end of an iteration. It is not an error condition. An exhausted iterator is a normal, expected state, and raising StopIteration from next() is the correct and documented way to indicate that state. Beginners sometimes try to avoid StopIteration by returning a special value instead, but this breaks the protocol and causes infinite loops in for statements and other consumers.

The StopIteration exception should be raised only from next(). If a generator function raises StopIteration directly, Python 3.7 and later convert it into a RuntimeError, because a StopIteration leaking out of a generator body is almost always a bug. If you need to signal the end of a generator, use a plain return statement, which the generator machinery translates into StopIteration internally.

When you are manually consuming an iterator with next(), you have two options for handling StopIteration. You can wrap the next() call in a try-except block and break or return when StopIteration is caught, or you can use the two-argument form of next(it, default), which returns default instead of raising StopIteration. The two-argument form is convenient when you have a natural sentinel value that cannot be confused with a valid item from the iterator.

The relationship between iterables and the protocol

An iterable is any object whose iter() method returns an iterator. The iterable itself does not need to implement next(). Its job is to serve as a factory for iterators, creating a fresh iterator each time iter() is called. This factory pattern is what enables multiple independent traversals of the same underlying data. Each call to iter() on a list produces a new list iterator that starts from index zero, and advancing one iterator does not affect any other iterator created from the same list. You will see this factory pattern implemented in detail when you learn about creating custom iterators in Python.

An iterator, by contrast, is any object that implements both iter() returning self and next() returning the next item or raising StopIteration. The iterator is both the factory and the product: calling iter() on it returns itself, and calling next() on it advances its internal state. This self-referential design means iterators are inherently single-use. Once exhausted, calling iter() returns the same exhausted iterator, and calling next() will immediately raise StopIteration again.

This distinction between iterable-as-factory and iterator-as-cursor is the key insight that makes the protocol feel natural once you internalize it. When you design a class that should support multiple iterations, you separate the data storage from the traversal logic. The class itself, acting as the iterable, stores the data and implements iter() to return a new instance of a separate iterator class. The iterator class stores only a reference to the data and a current position, implements iter() to return self, and implements next() to produce the next item. You will see this pattern in practice when you learn about the iter() function and how it creates iterators from any iterable.

The protocol and Python's data model

The iterator protocol is one of several protocols in Python's data model that are defined entirely by method names rather than by inheritance from a base class. Python does not require iterators to inherit from an Iterator abstract base class, though the collections.abc module provides one for type checking and for registering virtual subclasses. Any object that defines iter() and next() with the correct signatures and semantics is a valid iterator, regardless of its inheritance hierarchy.

This duck-typing approach to protocols is characteristic of Python's design philosophy. It means you can add iteration support to existing classes by implementing the two methods, without changing the class's inheritance tree or importing anything from the standard library. It also means that Python checks for protocol compliance at runtime rather than at class definition time. If your next() method has a bug that causes it to return None instead of raising StopIteration, you will not get an error until a for loop runs forever or a list() call exhausts memory.

The collections.abc.Iterator abstract base class is still useful. It provides a default iter() implementation that returns self and a subclasshook() that recognizes any class with both iter() and next() as a virtual subclass. This means isinstance(obj, Iterator) returns True for any object that follows the protocol, even if it does not explicitly inherit from Iterator. You can use this for defensive type checking in functions that need to distinguish between iterators and reusable iterables.

Why the protocol matters for performance

The iterator protocol enables lazy evaluation, which is the practice of computing values only when they are needed rather than computing them all in advance. A range object with a stop value of one billion implements the iterator protocol to compute each integer on demand, storing only the start, stop, and step internally. Iterating over such a range consumes constant memory regardless of the range's size, because only one integer exists in memory at any given moment.

This lazy model extends to file processing, network streams, database cursors, and sensor data. Any data source that can produce values sequentially can be wrapped in an iterator that reads or computes one value at a time. A program that processes a terabyte of log data can use an iterator-based pipeline that never holds more than a few kilobytes in memory. The alternative, reading the entire dataset into a list before processing, would be impossible on most machines.

The protocol also enables composition. Because every iterator presents the same interface, you can chain iterators together to build processing pipelines. The map() function applies a transformation to each item from an input iterator and returns a new iterator that yields the transformed values. The filter() function returns an iterator that yields only the items that pass a predicate. The zip() function combines multiple iterators into a single iterator of tuples. Each stage in the pipeline consumes items from the previous stage, processes them, and yields results to the next stage, all without materializing intermediate lists. The articles on Python generators and the iter() and next() functions later in this section show you how to build these pipelines with the full power of Python's iteration tools.