Write Memory-Efficient Python Code

Writing memory-efficient Python code means using less RAM to accomplish the same work. This matters when your program processes datasets larger than available memory, runs on a constrained server, or creates millions of objects.

Python objects carry overhead. An integer is not just 4 or 8 bytes, it is a full object with a reference count, a type pointer, and a variable-length value, typically 28 bytes for a small integer on a 64-bit system.

A list of a thousand integers stores a thousand 8-byte pointers plus list overhead plus the overhead of each integer object.

Understanding this overhead is the first step. The second step is knowing which tools and patterns reduce that overhead without sacrificing readability.

How Python objects use memory

Every Python object has a fixed-size header followed by variable data. The header includes a reference count and a pointer to the object's type. Even an empty object consumes memory for this header.

python

import sys
 
print(sys.getsizeof(0))
print(sys.getsizeof(""))
print(sys.getsizeof([]))
print(sys.getsizeof({}))

The output shows that an integer, an empty string, an empty list, and an empty dictionary all have non-zero sizes. The values vary by Python version, but on CPython 3.14 a small integer is 28 bytes, an empty string is 41 bytes, an empty list is 56 bytes, and an empty dictionary is 64 bytes.

sys.getsizeof() reports the shallow size: the memory the object itself occupies, not the objects it references. For a list, the shallow size is the array of pointers to its elements, not the elements themselves. To measure the deep size (the list plus all its elements), you need a recursive tool or a library like pympler.

python

import sys
 
nums = list(range(1000))
print(sys.getsizeof(nums))

This reports the size of the list's internal array: roughly 8 bytes per pointer times 1000 slots, plus the list object overhead. It does not include the size of the 1000 integer objects, each of which is 28 bytes. The deep memory cost of list(range(1000)) is the list overhead plus 1000 times the pointer size plus 1000 times the integer object size.

Generators: process data one item at a time

A generator produces values lazily. Instead of building a collection and returning it all at once, a generator yields one value at a time and pauses between yields. The memory cost is constant regardless of how many values the generator produces.

python

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()
 
for line in read_lines("large_file.txt"):
    process(line)

The read_lines generator reads one line, yields it, and reads the next line only when the loop asks for it. Only one line is in memory at a time. If large_file.txt is 2 GB, the memory usage stays at roughly the size of the longest line, not 2 GB.

Compare this with reading the entire file into a list:

python

lines = [line.strip() for line in open("large_file.txt")]

This version reads every line into memory before processing begins. The memory cost is proportional to the file size.

Generator expressions

A generator expression is the lazy equivalent of a list comprehension. Wrap the comprehension in parentheses instead of brackets:

python

sum(i * i for i in range(10_000_000))

This computes the sum without ever creating a list of ten million squares. Each square is computed on demand, added to the running total, and then discarded. The memory usage is constant.

The same computation with a list comprehension:

python

sum([i * i for i in range(10_000_000)])

This builds a list of ten million integers before sum() begins. The list alone occupies about 80 MB for the pointers, plus 280 MB for the integer objects. The generator version uses a few hundred bytes.

Generator expressions are most useful when chaining operations. Instead of creating intermediate lists at each step, chain generators so each element flows through the entire pipeline before the next element is produced.

python

nums = range(1_000_000)
squares = (x * x for x in nums)
evens = (s for s in squares if s % 2 == 0)
result = sum(evens)

No intermediate collection is materialized. Each number flows from nums through squares through evens into the sum accumulator, and then the next number starts.

Slots: reduce per-instance memory

Every Python object that does not define __slots__ stores instance attributes in a dictionary called __dict__. The dictionary overhead is significant for classes with many small instances.

python

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
points = [Point(i, i) for i in range(100_000)]

Each Point instance carries a dict that holds x and y. The dictionary overhead, roughly 184 bytes minimum, dominates the memory cost when the instance data is just two numbers. One hundred thousand Point instances without slots can consume tens of megabytes.

__slots__ replaces the per-instance __dict__ with a fixed-size array of slots, one per declared attribute.

python

class Point:
    __slots__ = ("x", "y")
 
    def __init__(self, x, y):
        self.x = x
        self.y = y
 
points = [Point(i, i) for i in range(100_000)]

Each Point instance now stores x and y in a compact C-level array instead of a dictionary. The memory savings are substantial, often 50 to 70 percent for simple data classes with many instances.

Slots comes with a tradeoff. You cannot add new attributes to a slotted instance after creation. The following raises AttributeError:

python

p = Point(1, 2)
p.z = 3

For data classes that represent fixed-shape records, this restriction is a benefit because it prevents accidental attribute creation. For classes that need dynamic attributes, slots is not appropriate.

Dataclasses with slots

Python's dataclasses module supports slots natively. Set slots=True in the decorator:

python

from dataclasses import dataclass
 
@dataclass(slots=True)
class Record:
    id: int
    name: str
    score: float

This combines the convenience of dataclasses with the memory efficiency of slots. The init, repr, and eq methods are generated automatically, and the instances do not carry a dict.

The array module: compact numeric storage

A Python list of numbers stores a pointer per element, and each element is a full Python integer or float object. For large arrays of homogeneous numeric data, the array module stores raw C-level values directly, eliminating the per-element object overhead.

python

import array
import sys
 
py_list = [i for i in range(10_000)]
arr = array.array("i", range(10_000))
 
print(sys.getsizeof(py_list))
print(sys.getsizeof(arr))

The list is an array of 10,000 pointers, 8 bytes each. The array.array("i") call is a contiguous block of 10,000 C integers, 4 bytes each, so the container itself is roughly half the size. The real difference is much larger once you count what those pointers reference: each integer in the list is a separate Python object of about 28 bytes, while the array stores raw values with no per-element objects at all.

The type code "i" specifies signed integer. Other useful codes are "d" for double-precision float, "f" for single-precision float, and "b" for signed char. The full list is in the array module documentation.

Operations on array objects are similar to list operations but return Python objects when you access individual elements:

python

arr = array.array("i", [1, 2, 3, 4, 5])
print(arr[0])
print(arr[0] * 2)
arr.append(6)

The element access creates a temporary Python integer, so the memory savings apply to storage, not to element access. This is the right tradeoff for most programs: you save memory where the data lives and pay the object-creation cost only when you need a specific value.

Streaming large files

Reading an entire file into memory works for small files but fails or thrashes for files larger than available RAM. Streaming processes the file in chunks.

python

CHUNK_SIZE = 64 * 1024
 
def process_large_file(path):
    with open(path, "rb") as f:
        while True:
            chunk = f.read(CHUNK_SIZE)
            if not chunk:
                break
            process(chunk)

Each chunk is a fixed-size bytes object that is overwritten on the next iteration. The memory usage is CHUNK_SIZE plus the processing overhead, regardless of file size.

For text files, reading line by line is often the most natural streaming pattern:

python

with open("large.csv") as f:
    header = next(f)
    for line in f:
        process(line)

Python's file object is itself a line iterator, and iterating with for line in f reads one line at a time. This works for files of any size.

For CSV processing with the csv module, pass the file object directly rather than reading all rows into a list:

python

import csv
 
with open("large.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        process(row)

Each row is a dictionary created for the current line and then discarded when the loop advances. The memory usage is bounded by the largest single row, not the file size.

Avoiding common memory traps

Trap 1: materializing intermediate lists

Chaining list operations creates a new list at each step.

python

data = [x for x in range(1_000_000)]
filtered = [x for x in data if x % 2 == 0]
doubled = [x * 2 for x in filtered]

Three lists exist simultaneously: data, filtered, and doubled. With a million integers each, this consumes significant memory.

The generator-based version uses constant memory:

python

data = range(1_000_000)
gen = (x * 2 for x in data if x % 2 == 0)
result = sum(gen)

Trap 2: holding references longer than needed

A variable that remains in scope keeps its referenced object alive. If a large dataset is bound to a variable after the program finishes using it, the memory is not freed.

python

def process_huge_file(path):
    data = open(path).read()      # huge string
    result = compute(data)
    # data is still in scope here
    return result

The fix is not to call del everywhere. Restructuring the work into smaller functions lets large objects fall out of scope naturally as soon as each function returns.

python

def load_data(path):
    return open(path).read()
 
def compute_result(data):
    result = compute(data)
    return result
 
def process_huge_file(path):
    data = load_data(path)
    result = compute_result(data)
    return result
    # data goes out of scope here

Trap 3: accidental reference cycles

Objects that reference each other, directly or indirectly, prevent Python's reference-counting garbage collector from freeing them. The cycle detector handles most cases, but in memory-constrained programs, breaking cycles explicitly with weakref can help.

python

import weakref
 
class Node:
    def __init__(self, value):
        self.value = value
        self.parent = None
        self.children = []
 
    def add_child(self, child):
        child.parent = weakref.ref(self)
        self.children.append(child)

A weakref.ref allows the parent reference without preventing garbage collection of the parent object. When the last strong reference to the parent is removed, the parent is freed even if children still hold weak references.

Measuring memory improvements

Before optimizing for memory, establish a baseline. tracemalloc tracks every allocation and produces a diff between two snapshots.

python

import tracemalloc
 
tracemalloc.start()
 
# Code before optimization
snapshot_before = tracemalloc.take_snapshot()
 
# Apply optimization here
# ... your optimized code ...
 
snapshot_after = tracemalloc.take_snapshot()
 
for stat in snapshot_after.compare_to(snapshot_before, "lineno")[:5]:
    print(stat)

The compare_to method returns statistics sorted by the size difference between the two snapshots. Positive values mean the after-snapshot used more memory. Negative values mean the optimization saved memory.

For a quick size check on a specific object, sys.getsizeof is sufficient. For tracking which parts of a program allocate the most memory, tracemalloc is the right tool.

The article on Python performance optimization covers the broader workflow of measuring, profiling, and optimizing. When memory is the bottleneck rather than speed, the techniques here are the right starting point. For loop-level speed improvements, the next article covers optimizing Python loops and iterations.