Read and Write Large Files in Python

The Python large files problem is one of the first scalability challenges you encounter as a programmer. The techniques that work beautifully for a ten-kilobyte configuration file, calling read to pull everything into memory as a single string or calling readlines to build a list of every line, can freeze your program or crash it with a MemoryError when applied to a ten-gigabyte log file. The file system does not care about file size; it stores a terabyte file with the same mechanisms it uses for a ten-byte file. The limitation is in your program's memory, and the solution is to process files in small, sequential pieces rather than loading them wholesale.

Python gives you several built-in tools for memory-efficient file processing, and none of them require third-party libraries or advanced language features. The same file object you have been using throughout this section supports chunked reading, line-by-line iteration, and buffered writes that keep memory usage constant regardless of file size. This article covers the patterns you need to read and write large files safely, building on the foundation from the articles on reading files in Python and writing files in Python.

Why naive reading fails on large files

To understand the problem, consider what happens when you call read with no arguments on a large file. Python asks the operating system for the file's entire content, the operating system delivers millions or billions of bytes, and Python stores them all in a single string in your program's memory. If the file is two gigabytes and your Python process has only one gigabyte of available memory, the allocation fails. The same thing happens with readlines, which builds a list of strings where each string is one line. A file with a hundred million short lines creates a list of a hundred million string objects, each with its own memory overhead, and the total memory usage far exceeds the file's disk size.

The solution is to never ask for the entire file at once. Instead, ask for a piece, process it, discard it, and ask for the next piece. This is called streaming or chunked processing, and Python's file object supports it natively. There are two streaming patterns: line-by-line iteration for text files where each line is a meaningful unit of work, and chunked reading with a fixed buffer size for binary files or text files where line boundaries are not the natural unit of processing.

Line-by-line iteration for text files

The for loop iterator pattern is the simplest and most memory-efficient way to process a large text file. When you write for line in file, Python reads a buffer of data from disk, splits it into lines, and yields one line at a time. The buffer size is managed internally and optimized for the operating system, and you never need to think about it. Each line is discarded after the loop body finishes with it, so memory usage stays constant.

python

error_count = 0
with open("server.log", "r", encoding="utf-8") as file:
    for line in file:
        if "ERROR" in line:
            error_count += 1
print(f"Found {error_count} error lines.")

This code could process a log file of any size, from a few kilobytes to hundreds of gigabytes, without ever using more than a small, fixed amount of memory. The logic is identical for both cases because the for loop abstracts away the buffering and chunking. This is the pattern you should reach for whenever your processing logic is line-oriented, which covers the vast majority of text file processing tasks: filtering, counting, transforming, searching, and extracting.

One important detail is that the for loop includes the trailing newline character on each line. If your processing logic concatenates lines or writes them to an output file, remember to handle the newline correctly. Calling strip removes it, and calling rstrip removes only trailing whitespace while preserving leading whitespace. If you are writing processed lines to an output file, you can write them as-is because the newline is already present and you do not need to add another.

Chunked reading for binary and non-line formats

For binary files or text files where lines are not the natural unit, use chunked reading with a fixed buffer size. The read method accepts an optional integer argument that specifies the maximum number of characters, or bytes in binary mode, to read in a single call. After each chunk is processed, calling read again continues from where the previous call left off. When the file is exhausted, read returns an empty string or empty bytes object, and the loop stops.

python

chunk_size = 8192
with open("large_data.bin", "rb") as file:
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            break
        process(chunk)

The chunk size controls the tradeoff between memory usage and system call overhead. A larger chunk size reads more data per system call, which is faster, but uses more memory per iteration. A smaller chunk size uses less memory but makes more system calls, which adds overhead. A value of 8192 bytes, or eight kilobytes, is a reasonable default that balances both concerns on most systems. For very large files on high-performance storage, you might increase this to 64 kilobytes or more. For embedded systems or memory-constrained environments, you might decrease it.

Chunked reading works whether the file is ten megabytes or ten terabytes. The loop structure is identical, and the memory usage is bounded by the chunk size regardless of file size. This pattern is the foundation of file copying, data transformation pipelines, network transfer utilities, and any program that moves or processes data without interpreting its content structure.

Writing large files efficiently

Writing large files uses the same principle in reverse: do not build the entire output in memory before writing it. Instead, generate and write the output in small pieces, letting Python buffer the writes and flush them to disk efficiently. The write method itself buffers data internally, and the operating system buffers disk writes as well, so calling write many times with small strings is not as inefficient as it might seem.

python

with open("output.csv", "w", encoding="utf-8") as outfile:
    outfile.write("id,name,score\n")
    for record in generate_records():
        line = f"{record.id},{record.name},{record.score}\n"
        outfile.write(line)

The generate_records function could be reading from a database, processing a large input file, or generating data algorithmically. As long as it yields records one at a time rather than building a list of all records, the memory usage stays constant. The write calls accumulate in Python's internal buffer, and when the buffer fills up, Python flushes it to disk in a single efficient write system call. This automatic buffering means you can write each line as it is generated without worrying about the performance cost of many small disk writes.

For binary output, the same pattern applies: write chunks as you generate them rather than accumulating everything in a bytes object. If you are copying a file, read a chunk from the source and immediately write it to the destination. The chunk size acts as the window of data in flight, and the total memory usage is two chunks, one for reading and one for writing, regardless of file size.

Practical large file processing patterns

Combining chunked reading with chunked writing is the standard pattern for file transformation and copying. The source file is opened for reading, the destination file is opened for writing, and data flows from one to the other in chunks. This pattern processes files of any size because it never holds more than a chunk of data in memory at any time.

For CSV processing, the csv module's reader and writer objects work directly with file objects and do not load the entire file into memory. Wrapping a file opened for reading in a csv.reader gives you an iterator that yields one row at a time, and wrapping a file opened for writing in a csv.writer lets you write one row at a time. The memory usage per row is proportional to the row's size, not the file's size, which makes csv suitable for very large datasets.

For JSON, the standard json.load function reads the entire file into memory, which fails on large files. For line-delimited JSON, where each line is a complete JSON object, you can iterate over the file line by line and parse each line individually. This format, sometimes called JSON Lines or NDJSON, is specifically designed for the kind of streaming processing that Python's for loop pattern enables.

The patterns in this article close out the practical file handling skills you need for most real-world Python programs. Combined with the path handling, mode selection, and error handling from the earlier articles in this section, you can now read from files, write to files, navigate the file system, handle errors gracefully, and process datasets of any size without exhausting memory. The remaining articles in this section cover common mistakes to avoid and a capstone project that ties everything together.