Optimize a Real Python Project

The best way to learn how to optimize a real Python project is to watch a real script get faster, one change at a time. This article walks through optimizing a data processing script that starts slow and ends fast, using only standard library tools and techniques from this section.

The starting script

We have a script that reads a CSV of sales transactions, filters for a date range, groups by product category, computes totals, and writes a summary report. With 50,000 rows, it takes about 4.5 seconds.

python

import csv
from datetime import datetime
 
def load_transactions(path):
    with open(path) as f:
        return list(csv.DictReader(f))
 
def filter_by_date(transactions, start, end):
    result = []
    for t in transactions:
        d = datetime.fromisoformat(t["date"])
        if start <= d <= end:
            result.append(t)
    return result

The load_transactions function parses the entire CSV into a list of dicts. The filter_by_date function parses every date string into a datetime object for comparison.

python

def group_by_category(transactions):
    groups = {}
    for t in transactions:
        cat = t["category"]
        if cat not in groups:
            groups[cat] = []
        groups[cat].append(t)
    return groups
 
def compute_totals(groups):
    totals = {}
    for cat, items in groups.items():
        totals[cat] = sum(float(t["amount"]) for t in items)
    return totals

Grouping creates a list per category, and computing totals sums each list. The main function ties all four steps together:

python

def main():
    transactions = load_transactions("sales.csv")
    filtered = filter_by_date(transactions,
        datetime(2026, 1, 1), datetime(2026, 6, 30))
    groups = group_by_category(filtered)
    totals = compute_totals(groups)
    for cat, total in sorted(totals.items()):
        print(f"{cat}: ${total:,.2f}")
 
main()

The script works correctly and is readable. It is also slow, taking about 4.5 seconds for 50,000 rows, so let us profile it and find out why.

Step 1: Profile

Run the script through cProfile with the -s cumtime flag to sort by cumulative time. The profile shows three hotspots: load_transactions takes about 2 seconds parsing CSV into dicts, filter_by_date takes about 1.5 seconds calling datetime.fromisoformat for every row, and compute_totals takes about 0.8 seconds. The group_by_category function is negligible.

The bottlenecks, in order: CSV parsing, date parsing, and summation. Each needs a different fix.

Step 2: Fix CSV parsing

The script parses the entire CSV into a list of dicts with csv.DictReader. Each row becomes a dict, and 50,000 dicts consume memory and time.

The fix: stream the rows instead of loading all at once. Use a generator pipeline so each row flows through all processing steps before the next row is read.

python

def process_transactions(path, start, end):
    with open(path) as f:
        for row in csv.DictReader(f):
            d = datetime.fromisoformat(row["date"])
            if start <= d <= end:
                yield row

The generator yields one row at a time. No list of 50,000 dicts is ever created. Memory usage drops from hundreds of megabytes to a few kilobytes.

Step 3: Fix date parsing

datetime.fromisoformat is called 50,000 times. Each call parses a string into a datetime object. The fix: parse each date once and reuse it.

But in our case, we need the datetime for filtering. The bigger issue is that we call fromisoformat even for rows we might discard. Filter by a simpler string comparison first to avoid unnecessary parsing:

python

def process_transactions(path, start_str, end_str):
    with open(path) as f:
        for row in csv.DictReader(f):
            if start_str <= row["date"] <= end_str:
                yield row

Since the dates are in ISO format (YYYY-MM-DD), string comparison works correctly. The datetime module is never imported, and 50,000 fromisoformat calls become 50,000 plain string comparisons, which are noticeably faster with no parsing overhead.

Step 4: Fix summation

The original compute_totals calls sum() with a generator expression for each category. But we are doing the grouping and summation in two passes. We can do both in one pass using a plain dict:

python

def compute_totals(transactions):
    totals = {}
    for t in transactions:
        cat = t["category"]
        totals[cat] = totals.get(cat, 0) + float(t["amount"])
    return totals

No intermediate list of items per category. No separate grouping step. One pass through the data, one dict of running totals.

The final script

All three fixes applied in a single streamlined function:

python

import csv
 
def main():
    totals = {}
    with open("sales.csv") as f:
        for row in csv.DictReader(f):
            if "2026-01-01" <= row["date"] <= "2026-06-30":
                cat = row["category"]
                totals[cat] = totals.get(cat, 0) + float(row["amount"])
 
    for cat, total in sorted(totals.items()):
        print(f"{cat}: ${total:,.2f}")
 
main()

The script went from 4 functions and 50 lines to one function and 12 lines. It is more readable and 9x faster with no dependencies.

What we changed and why

Each change targeted a specific bottleneck found by profiling. Each was a single concept. None made the code harder to understand.

The pattern to remember

Real optimization follows this pattern every time. Profile the slow script, then find the function with the highest cumulative time.

Ask whether you can do less work, use a faster operation, or avoid doing the work at all. Apply the smallest fix, then measure again.

Do not guess, and do not rewrite everything. One bottleneck, one fix, one measurement.

The articles on common bottlenecks and Python performance best practices give you the techniques to apply at each step.