Use the with Statement in Python

When you use the with statement in Python, you activate a context manager that guarantees resource cleanup. Learning to use with correctly is as fundamental as learning a for loop. It appears in every Python codebase that reads files, acquires locks, manages database transactions, or handles any resource that needs guaranteed cleanup. The syntax is simple: the keyword with, followed by an expression that evaluates to a context manager, optionally followed by the as keyword and a variable name, followed by a colon and an indented block. Despite its simplicity, the with statement has nuances around multiple context managers, exception handling, and variable scoping that are worth understanding in detail.

The previous article on Python context managers explained established that context managers implement a protocol with enter and exit methods, and that the with statement calls enter before the block and exit after it. This article focuses on the practical usage patterns: how to use with for files, locks, and other common resources, how to combine multiple context managers, and how the as clause interacts with variable scope and exception handling. If you have written Python for any length of time, you have used with for file handling. But you may not have explored its full range of applications.

The relationship between decorators and context managers is worth keeping in mind as you read. Decorators add behaviour around function calls and run every time the function is invoked. Context managers add behaviour around code blocks and run once per with statement. A function decorated with a timing decorator adds timing to every call. A with block around a timing context manager times a specific block of code. Both patterns have their place, and choosing between them depends on whether the behaviour belongs to the function's identity or to a specific usage of the function.

The basic pattern for files

The most common place to use the with statement in Python is opening files. The open function, covered in depth in Python file handling explained, returns a file object that implements the context manager protocol. The enter method returns the file object itself, which is usually what you bind with the as clause. The exit method closes the file. Here is the canonical pattern for reading a file:

python

with open("data.txt", "r") as file:
    content = file.read()
    print(f"Read {len(content)} characters")

The file is opened before the block runs, and it is closed when the block ends. The variable named file is bound by the as clause and is available throughout the block. After the block, the file is closed, and the variable still exists but refers to a closed file object. Accessing the variable after the block does not raise an error, but any attempt to read or write through it will raise a ValueError because the underlying file descriptor has been released.

The with statement replaces the older pattern of opening a file, using it, and closing it in a finally block. The older pattern required three separate steps and was easy to get wrong: forgetting the finally block, putting the close in the wrong place, or accidentally closing the file before all reads were complete. The with statement makes the correct pattern the only pattern, which is why it is universally recommended for file handling.

Here is the same operation for writing a file, which follows the identical pattern. The only difference is the mode string:

python

lines = ["first line\n", "second line\n", "third line\n"]
with open("output.txt", "w") as file:
    file.writelines(lines)

The write mode opens the file for writing, creating it if it does not exist and truncating it if it does. The writelines call writes all the strings in the list to the file. When the block ends, the file is flushed and closed. Any data that was buffered but not yet written to disk is written during the close, which the exit method handles automatically.

Working with multiple context managers

Python allows multiple context managers in a single with statement, separated by commas. This is the preferred way to work with multiple resources that all need to be open at the same time. The comma-separated form is equivalent to nesting with statements, but it is flatter and more readable. Here is an example that copies content from one file to another using a single with statement with two context managers:

python

with open("source.txt", "r") as src, open("dest.txt", "w") as dst:
    for line in src:
        dst.write(line)

Both files are opened before the block runs. The source file is opened for reading and the destination file for writing. The block reads each line from the source and writes it to the destination. When the block ends, both files are closed. If an exception occurs during the copy, both files are still closed because the with statement guarantees that every context manager's exit method runs, even if an earlier exit method raised an exception.

The order of context managers in a comma-separated with statement matches the order of nesting if you wrote them as separate with blocks. The first context manager's enter runs first, and its exit runs last. In the file copy example, the source file is opened first and closed last. The destination file is opened second and closed first. This ordering is consistent with how nested with statements behave and ensures that resources are released in the reverse order of acquisition, which is the correct pattern for most resource management.

You can also nest with statements explicitly when the inner context manager depends on a value from the outer one. This pattern is less common but necessary when the second context manager cannot be created until the first has produced a value. Here is an example where the inner context manager uses data read from the outer file to construct a path:

python

with open("config.txt") as config:
    output_path = config.readline().strip()
    with open(output_path, "w") as output:
        output.write("Processed")

The outer with opens the configuration file and reads the first line to get the output path. The inner with opens that output path for writing. Both files are closed when their respective blocks end. The nesting makes the dependency explicit: the output file cannot be opened until the configuration file has been read. If both files were opened in a single comma-separated with, Python would try to open them simultaneously, which would fail because the output path is not yet known.

Using with for locks and synchronization

The threading module's Lock class is a context manager, and you should also use the with statement in Python code that acquires and releases locks. Acquiring a lock inside a with block guarantees that the lock is released when the block ends, even if the block raises an exception or the thread is interrupted. This prevents the deadlocks that occur when a lock is acquired but never released because an exception skipped the release statement.

Here is the pattern for protecting a shared data structure with a lock:

python

import threading
 
counter = 0
lock = threading.Lock()
 
def increment():
    global counter
    with lock:
        current = counter
        counter = current + 1

The lock is acquired before the block runs, and it is released when the block ends. If multiple threads call increment simultaneously, only one thread holds the lock at a time. The other threads block at the with statement until the lock is released. The critical section, reading and updating the counter, is protected from concurrent access.

The with statement for locks replaces the older pattern of calling acquire and release manually, which required a try-finally block to ensure release. The with statement is shorter, and it prevents the bug where a programmer forgets the finally block and the lock is never released. For reentrant locks, the same pattern works correctly because the Lock class implements the context manager protocol consistently.

The as clause and variable scoping

The as clause in a with statement binds the value returned by the context manager's enter method to a variable. This variable is available throughout the with block and continues to exist after the block ends. This is different from some other languages where variables bound in a resource block go out of scope when the block ends. In Python, the variable persists, but the resource it refers to has been closed or released.

For file objects, the variable after the with block refers to a closed file. You can inspect its name, mode, and closed status, but you cannot read or write through it. For locks, the variable after the with block refers to a released lock. You can still call its locked method (which will return False), and you can acquire it again in another with block.

The persistence of the as variable after the block is a design choice that prioritizes simplicity over strict scoping. It means you can inspect the context manager after the block to check its final state, which is useful for debugging. It also means you do not need to declare the variable before the with statement, which keeps the syntax clean. The tradeoff is that you can accidentally use the variable after the block and get a confusing error about a closed resource, but this mistake is easy to catch and fix.

The with statement in real-world code

Beyond files and locks, you use the with statement in Python for database access, network communication, and testing. Database drivers like sqlite3 and psycopg2 provide connection objects that are context managers. Using a connection inside a with block ensures that transactions are committed on success and rolled back on exception. The pattern is identical to the file and lock patterns, which is the beauty of the context manager protocol: every resource, regardless of its nature, uses the same syntax for guaranteed cleanup.

Network libraries provide session objects that are context managers. An HTTP client session acquired inside a with block guarantees that the underlying connection pool is closed when the block ends, preventing socket leaks in long-running applications. Testing frameworks provide context managers for temporary directories, captured output, and monkey-patched attributes. Each of these follows the same enter-exit protocol and uses the same with syntax.

Once you are comfortable using context managers, the next article covers creating custom context managers in Python. You will learn to implement the enter and exit methods on your own classes, turning any setup-and-teardown pattern into a reusable context manager that works with the with statement.