Optimize File and Data Processing in Python

File and data processing bottlenecks usually come from I/O, not CPU. The disk or network is slow, and your Python code spends most of its time waiting. Optimizing file processing is about reducing how much data you read and how you read it.

The single biggest mistake is loading an entire file into memory when you only need it one piece at a time. A 2 GB log file does not need to be in RAM to count error lines. Streaming processes the file in chunks and keeps memory usage constant.

Read in chunks, not all at once

Reading an entire file with f.read() is simple but dangerous for large files. Chunked reading processes a fixed-size block at a time.

python

CHUNK = 64 * 1024
 
with open("large_file.bin", "rb") as f:
    while chunk := f.read(CHUNK):
        process(chunk)

The walrus operator (:=) reads a chunk and checks if it is empty in one expression. Memory usage stays at CHUNK bytes regardless of file size.

Line-by-line for text files

Python's file object is an iterator over lines. Use it directly instead of calling readlines(), which creates a list of all lines in memory.

python

with open("large.log") as f:
    for line in f:
        if "ERROR" in line:
            print(line.rstrip())

Each line is read and discarded as the loop advances. A 10 GB log file uses memory proportional to the longest line, not the file size.

For CSV files, pass the file object to the csv module:

python

import csv
 
with open("data.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        process(row)

Binary is faster than text

Text mode decodes bytes into strings. If you do not need decoded text, binary mode skips this step and is measurably faster.

python

# Binary: fast, no decoding.
with open("source.bin", "rb") as src, open("dest.bin", "wb") as dst:
    while chunk := src.read(64 * 1024):
        dst.write(chunk)
 
# Text: slower, decodes and encodes.
with open("source.txt") as src, open("dest.txt", "w") as dst:
    for line in src:
        dst.write(line)

For file copy operations, use shutil.copyfileobj which handles buffering efficiently. For data processing where every byte matters, binary mode is the right choice.

Buffer size matters

Python buffers I/O by default: typically 8 KB for binary files. Larger buffers mean fewer system calls, which improves throughput for large sequential reads.

python

with open("large.bin", "rb", buffering=256 * 1024) as f:
    while chunk := f.read(256 * 1024):
        process(chunk)

The buffering parameter sets the internal buffer size. Matching the read size to the buffer size avoids double-copying. For modern SSDs, 64 KB to 256 KB is a good range. Beyond that, returns diminish.

Handle formats efficiently

JSON parsing is expensive. If you process a large JSON array, do not load it all at once. Use a streaming parser like ijson.

python

import ijson
 
with open("large.json", "rb") as f:
    for record in ijson.items(f, "item"):
        process(record)

Each record is yielded one at a time. The file is never fully materialized. This works for multi-gigabyte JSON files that would crash json.load().

For tabular data, CSV is simple but slow for large datasets. If you process the same file repeatedly, consider Parquet or a database. The time spent converting formats often pays back quickly.

Use the right tool for the data shape

Not all data needs Python. When you need to filter, count, or aggregate structured text, command-line tools are faster because they are written in C and optimized over decades.

text

grep "ERROR" large.log | wc -l
cut -d',' -f3 data.csv | sort | uniq -c | sort -rn | head -10

When Python is the right tool, let built-in functions do the heavy work. The sum, max, min, sorted, and collections.Counter functions are implemented in C and run faster than manual loops.

python

from collections import Counter
 
with open("access.log") as f:
    ips = (line.split()[0] for line in f)
    top = Counter(ips).most_common(10)
 
print(top)

The generator expression (line.split()[0] for line in f) extracts IPs one at a time. Counter counts them in C. No intermediate list is created.

For the broader performance picture, the article on Python performance optimization covers how I/O bottlenecks fit into the profiling workflow. When processing speed is the problem rather than I/O, the article on optimizing Python loops covers the CPU side.