Generator Expressions in Python

List comprehensions are one of Python's most beloved features: concise, readable, and expressive. They let you transform and filter a sequence in a single line of code. But every list comprehension builds a complete list in memory. For ten items, that is irrelevant. For ten million items, it is a problem. Generator expressions solve this by providing the same comprehension syntax but with lazy evaluation, wrapped in parentheses instead of square brackets. The result is a generator object that produces values on demand, one at a time, without ever building a list.

If you have worked with Python generators and understand that they produce values lazily, a generator expression is simply a shorthand for a generator function that would consist of a single for loop yielding transformed items. Instead of defining a named function with def and yield, you write the transformation inline using syntax that looks exactly like a list comprehension, but with round parentheses instead of square brackets.

The practical benefit is immediate and measurable. Replacing a list comprehension that processes a million items with a generator expression can reduce peak memory usage from tens of megabytes to a few kilobytes. The generator expression itself is not inherently faster, each value still needs to be computed, but it avoids the allocation and garbage collection of a large list, and it starts producing results immediately rather than waiting for the entire list to be built.

Generator expression syntax

A generator expression looks identical to a list comprehension except for the enclosing brackets. Where a list comprehension uses [expr for var in iterable], a generator expression uses (expr for var in iterable). The expression can include filtering with an if clause and can iterate over multiple nested sources, just like a list comprehension. The key difference is that the result is a generator iterator, not a list:

python

squares_list = [x * x for x in range(10)]
squares_gen = (x * x for x in range(10))

The variable squares_list is a list of ten integers, all computed and stored in memory. The variable squares_gen is a generator object that will compute and yield each square only when asked. You can pass squares_gen to a for loop, to the list() constructor, or to any function that accepts an iterable, and the behaviour is identical to using the list comprehension, except for the memory profile.

When a generator expression is the only argument to a function call, you can omit the extra set of parentheses around it. Instead of writing sum((x * x for x in range(10))), you can write sum(x * x for x in range(10)). This syntactic convenience makes generator expressions especially clean as arguments to built-in functions like sum, min, max, any, and all:

python

readings = [3.2, -1.5, 4.8, 0.0]
words = ["hello", "world", "python"]
 
total = sum(x * x for x in range(1_000_000))
has_negative = any(val < 0 for val in readings)
largest = max(len(word) for word in words)

In each of these cases, the generator expression feeds values into the function one at a time. The function accumulates the result it needs, a sum, a boolean, or a maximum, and the generator never materializes a list of intermediate values. For the sum of a million squares, the generator expression uses a few hundred bytes of memory, while the list comprehension would use roughly eight megabytes just for the list of integers.

Generator expressions versus generator functions

A generator expression is the most compact way to create a generator, but it is limited to a single expression with optional filtering. If your value-production logic requires multiple steps, conditional branching, complex state, or try-finally cleanup, you need a generator function with def and yield. The article on creating generator functions with yield walks through each of these patterns in detail.

Generator expressions are especially well suited as inline arguments to functions. When you find yourself writing a one-line generator function whose body is a single for loop with yield, consider whether a generator expression would be clearer. A function named squares(n) that contains for i in range(n): yield i * i can be replaced with the generator expression (i * i for i in range(n)) passed directly where needed.

Generator functions have one important advantage over generator expressions: they have a name. When you use a generator expression inline, the reader must parse the expression to understand what values it produces. When you assign a generator expression to a variable with a descriptive name, or use a named generator function, the name documents the intent. For transformations used in multiple places, a named generator function is almost always preferable to duplicating a generator expression across the codebase.

Filtering with generator expressions

The filtering clause in a generator expression works exactly as it does in a list comprehension. An if clause at the end of the expression filters out values for which the condition is false, before the main expression is applied:

python

even_squares = (x * x for x in range(100) if x % 2 == 0)

This generator yields the squares of even numbers from 0 to 98. The if clause is evaluated first: if x is odd, the iteration skips to the next value without computing x * x. This is both memory-efficient and computation-efficient, because you never calculate squares for values that will be discarded.

You can chain multiple generator expressions together by nesting them or by passing the output of one as the input to another. A pipeline that filters, transforms, and aggregates can be expressed as a sequence of generator expressions, each feeding into the next:

python

numbers = range(1000)
evens = (x for x in numbers if x % 2 == 0)
squared = (x * x for x in evens)
sum_of_squares = sum(squared)

Each generator expression is evaluated lazily. When sum() requests the first value, squared asks evens for a value, which asks numbers for a value. Only when an even number is found does the square get computed and passed to sum(). The entire pipeline runs in a single pass, with no intermediate lists created at any stage. You can add, remove, or reorder stages without changing the memory profile of the pipeline.

Common pitfalls with generator expressions

The most common mistake with generator expressions is trying to reuse them. Like all generators, a generator expression is an iterator that can only be consumed once. After you iterate over it, the generator is exhausted, and any further attempt to iterate produces no values:

python

gen = (x * x for x in range(5))
print(list(gen))
print(list(gen))

The first call to list() consumes the generator and returns [0, 1, 4, 9, 16]. The second call returns an empty list because the generator is already exhausted. If you need to iterate over the values multiple times, either materialize them into a list or tuple, or recreate the generator expression each time.

Another pitfall is using a generator expression where you need random access or length checking. You cannot index into a generator, and len() does not work on generators. If your algorithm needs to access the tenth item directly or check how many items there are before processing, a generator expression is the wrong tool. Use a list comprehension instead, and accept the memory cost as the price of random access.

A subtler pitfall involves closing over loop variables in generator expressions. Like list comprehensions, generator expressions have their own local scope in Python 3, so the loop variable does not leak into the surrounding scope. However, if a generator expression references a variable from an enclosing loop, it captures the variable by name, not by value. If the enclosing loop variable changes before the generator is consumed, the generator sees the changed value. This is the same late-binding behaviour that affects closures and lambda expressions in Python, and the fix is the same: capture the value in a default argument or use a separate function to create a new scope.

Performance considerations

Generator expressions are not faster than list comprehensions for small datasets. The overhead of yielding values one at a time, with function-call semantics on each next() invocation, can make generator expressions slightly slower than list comprehensions when the entire dataset fits comfortably in memory. The benefit of generator expressions is not raw speed but memory efficiency and the ability to begin processing before all values are computed.

For the common case of passing a transformed sequence to an aggregating function like sum(), a generator expression is almost always the right choice. It avoids allocating a list that exists only to be immediately consumed and discarded. For interactive use in a Python REPL where you want to inspect intermediate results, a list comprehension is more convenient because you can see all the values at once.

The memory savings of generator expressions become dramatic when the input iterable is large and the output is consumed by an aggregating function. Summing the squares of the first ten million integers with a generator expression uses negligible memory. Summing them with a list comprehension requires storing ten million integers in a list, which occupies roughly eighty megabytes on a typical Python installation. The generator expression version also starts producing partial sums immediately, while the list comprehension version pauses while the entire list is built.