Python Generators Explained

Python generators are one of the language's most powerful features for writing memory-efficient, readable code that processes data one item at a time. If you have worked through the earlier articles on Python iterables and iterators and the iterator protocol, you already know that iterators are objects that track position and produce values on demand. A generator is a concise way to create an iterator by writing what looks like an ordinary function but uses the yield keyword instead of return. Every time you call next() on a generator, the function runs until it hits yield, hands a value back, and then freezes in place, preserving all its local variables for the next call.

The result is that you can write code that reads as a straightforward sequence of steps, processing one item, then the next, then the next, without ever building a giant list in memory. This lazy evaluation model is what makes generators indispensable for working with large datasets, streaming data from APIs, reading enormous files line by line, and building multi-stage data pipelines that transform values as they flow through. The same generator that produces ten values for a unit test can produce ten million values in production without changing a single line, because the generator never stores more than the current value at any moment.

Understanding generators also changes how you think about Python's memory model. Before generators, the natural approach to processing a sequence of values was to build a list, process the list, and discard it. That works until the list does not fit in memory. Generators replace the list with a pipeline: one stage reads raw data, the next stage parses it, the next stage filters it, and the final stage writes results. Each stage is a generator that yields one item at a time, and the entire pipeline consumes constant memory regardless of how much data flows through it.

How a generator function works

A generator function looks almost identical to a regular function. You define it with def, you give it a name, and you write a body of code. The difference is that instead of ending with a return statement that sends a final value back and destroys the function's local state, a generator function contains at least one yield statement. The yield keyword tells Python to pause the function, hand the yielded value to the caller, and remember exactly where it was so it can resume later.

When you call a generator function, Python does not execute the function body. Instead, it creates and returns a generator iterator object. This object implements the iterator protocol, iter() returns self and next() runs the function body until the next yield, without you writing either method. The function body runs only when you call next() on the generator object, either explicitly or implicitly through a for loop.

Consider this minimal generator that produces the numbers zero through four:

python

def simple_gen():
    for i in range(5):
        yield i
 
g = simple_gen()
print(next(g))
print(next(g))

The first call to next(g) runs the function body from the start. The for loop assigns 0 to i, and yield i pauses the function, returning 0. The second call to next(g) resumes the function right after the yield, with i still in scope and the for loop ready for its next iteration. The loop assigns 1 to i, yield pauses again, and 1 is returned. This continues until the loop finishes and the function body ends, at which point the generator automatically raises StopIteration, just like any other iterator.

What makes this remarkable is how little code it takes. Writing the equivalent as a custom iterator class would require defining iter() and next(), managing a position counter, and handling StopIteration explicitly. The generator does all of that automatically, letting you focus on the logic of what values to produce rather than the mechanics of how to produce them.

Generators and memory efficiency

The primary reason to choose a generator over a list is memory. A list that holds ten million integers occupies roughly eighty megabytes of RAM. A generator that produces the same ten million integers one at a time occupies a few hundred bytes, enough to store the current integer and the generator's internal state. This difference is not a minor optimization; it is the difference between a program that runs on a laptop and a program that crashes with a MemoryError.

This efficiency comes from lazy evaluation. The generator does not compute its next value until you ask for it. If you only need the first thousand values from a generator that could produce a million, you only pay for the thousand you consume. The remaining nine hundred and ninety-nine thousand values are never computed and never stored. This pay-as-you-go model is especially valuable when the values are expensive to compute or when the total number of values is unknown in advance, which is often the case when reading from a network socket, a database cursor, or a live data feed.

File processing is the canonical example of generator-powered memory efficiency. Reading an entire file into a list with readlines() loads every line into memory at once. A one-gigabyte log file becomes a one-gigabyte list. Iterating over the file object directly, which is itself a generator-like iterator, reads one line from disk at a time and never holds more than that single line in memory. The same principle applies to any data source that can be consumed sequentially: you wrap it in a generator, process one item at a time, and let the operating system handle the buffering.

Building data pipelines with generators

Because generators are iterators, you can chain them together to build processing pipelines where each stage consumes from the previous stage and yields to the next. This is the functional programming model applied to Python iteration: you compose small, focused generators rather than writing one large function that does everything.

Imagine you need to process a large file of user records, filter out inactive users, extract email addresses, and convert them to lowercase. With lists, each intermediate step would create a new list in memory. With generators, each step is a generator that yields one item at a time:

python

def read_users(filename):
    with open(filename) as f:
        for line in f:
            name, email, active = line.strip().split(",")
            yield {"name": name, "email": email, "active": active == "true"}
 
def active_users(users):
    for user in users:
        if user["active"]:
            yield user
 
def extract_emails(users):
    for user in users:
        yield user["email"].lower()

Each of these functions is a generator that does one small job. You compose them by passing the output of one as the input to the next. The entire pipeline runs in a single pass over the file, and no intermediate list is ever created. You can test each stage independently with a small hand-crafted input, then deploy the pipeline against a production file of any size. This composability is one of the main reasons experienced Python developers reach for generators when designing data processing code.

Generators versus custom iterator classes

Before generators were added to Python in version 2.2, every custom iterator had to be written as a class with iter() and next() methods. Generators did not replace custom iterator classes entirely, but they made them unnecessary for the vast majority of use cases. Any time your iteration logic can be expressed as a linear sequence of steps, a generator function is shorter, clearer, and less error-prone than the equivalent class.

There are still situations where a custom iterator class is the right choice. If your iterator needs to expose additional methods beyond next(), a generator object only provides send(), throw(), and close() in addition to the standard iterator protocol. If your iteration state is complex and branching, a class with named attributes can be easier to debug than a generator whose state is scattered across local variables and the current yield point. If you need to reset an iterator, a class can provide a reset method, while a generator must be recreated from scratch.

The article on Python iterators versus generators later in this section provides a detailed comparison with decision criteria. For now, the rule of thumb is: if you can describe your value-production logic in a few sentences of plain English, write it as a generator. If the logic requires complex state management, branching control flow that is hard to follow, or a public API beyond iteration, reach for a class.

The relationship between yield and return

In a regular function, return does two things: it provides a value to the caller, and it terminates the function, discarding all local state. In a generator function, yield provides a value to the caller but suspends the function rather than terminating it. The local variables, the instruction pointer, and the entire execution context are preserved so that the function can resume exactly where it left off.

A generator can also use a bare return statement, with no value, to terminate early. The return causes the generator to raise StopIteration, just as reaching the end of the function body does. Starting in Python 3.3, you can also write return with a value inside a generator, but the value is not directly accessible to the caller. Instead, it becomes the value attribute of the StopIteration exception, which is primarily useful when the generator is being driven by another generator using yield from.

The yield from statement, added in Python 3.3, lets one generator delegate to another. Instead of writing a loop that iterates over a sub-generator and yields each value individually, you write yield from sub_generator, and Python handles the iteration internally. This is especially useful for building recursive generators, like a generator that walks a directory tree and yields file paths, delegating to itself for each subdirectory it encounters. The next article in this section covers creating generator functions with yield in step-by-step detail.