Profile Python Performance Bottlenecks with `cProfile`

Python cProfile is the standard library's built-in profiler. It records every function call your program makes, counts how many times each function is called, and measures the total time spent in each function.

After your program finishes, you can sort and filter the results to find exactly which parts of the code dominate the runtime.

A profiler answers the most important performance question: "where should I spend my effort?" Without a profiler, you guess. With a profiler, you know.

The difference is often between optimizing a function that runs once at startup and optimizing a function that runs millions of times in a hot loop.

cProfile is distinct from timeit, which benchmarks a single small expression. cProfile looks at the whole program instead.

Use timeit when you have two candidate implementations, and use cProfile when you do not yet know which function to focus on. The article on measuring Python performance with timeit covers micro-benchmarks in detail.

How cProfile collects data

When you enable cProfile, it hooks into Python's function call machinery. Every time a function is entered or exited, cProfile records the event. It tracks two key metrics for each function:

• tottime: total time spent executing the function's own code, not including time in functions it called.
• cumtime: cumulative time, which includes the function's own code plus all the functions it called, directly or indirectly.

The distinction matters. If main() calls parse_file() and parse_file() calls json.loads(), the cumtime for main() includes all three, while the tottime for main() only includes code directly inside main().

This lets you tell whether a function is slow because it does heavy work itself or because it calls other slow functions.

cProfile also counts the number of calls to each function, including recursive calls. A function called millions of times with a small per-call cost can dominate runtime just as much as a function called once with a large cost.

Running cProfile the simple way

The quickest way to profile is from the command line. Replace python my_script.py with python -m cProfile my_script.py and the profiler prints a sorted table after the script finishes.

text

python -m cProfile my_script.py

The default output sorts by function name, which is rarely useful. Use the -s flag to sort by a meaningful column:

text

python -m cProfile -s cumtime my_script.py

The cumtime sort puts the functions with the highest cumulative time at the top, which is usually where your bottleneck lives. Other useful sort keys are tottime, calls, and ncalls.

The command-line approach works well for quick checks on scripts that run and exit. For more control, embed the profiler directly in your code.

Embedding cProfile in your code

When you need to profile a specific section of a larger program, enable the profiler around just that section and save the results for later analysis.

python

import cProfile
import pstats
import io
 
def work():
    total = 0
    for i in range(1_000_000):
        total += i
    return total
 
profiler = cProfile.Profile()
profiler.enable()
result = work()
profiler.disable()
 
stream = io.StringIO()
stats = pstats.Stats(profiler, stream=stream)
stats.sort_stats("cumtime")
stats.print_stats(10)
print(stream.getvalue())

The output is a table with columns for the number of calls, total time, time per call, cumulative time, and function name. Each row is one function.

The pstats.Stats object holds the profiling data and provides methods for sorting, filtering, and printing. sort_stats("cumtime") orders by cumulative time descending. print_stats(10) shows only the top ten rows, which keeps the output manageable for large programs.

Reading the profiler output

The profiler table has these columns:

The most important columns are cumtime and tottime. Look at cumtime first to find the overall bottleneck. Then look at the functions called by that bottleneck and check their tottime to see which one does the actual work.

python

def fast():
    return sum(range(1000))
 
def slow():
    total = 0
    for i in range(100_000):
        total += i ** 0.5
    return total
 
def main():
    fast()
    slow()

Nothing here is profiled yet. Profiling main() shows how fast and slow actually compare once the program runs, instead of guessing from how the code looks:

python

import cProfile
import pstats
import io
 
profiler = cProfile.Profile()
profiler.enable()
main()
profiler.disable()
 
stream = io.StringIO()
stats = pstats.Stats(profiler, stream=stream)
stats.sort_stats("cumtime")
stats.print_stats()
print(stream.getvalue())

The output will show slow with a higher cumtime and tottime than fast. The main function will have the highest cumtime because it called both, but its tottime will be near zero because it does no work of its own. This pattern is common: the entry-point function has the highest cumtime because it calls everything else, but the real work happens in functions further down the call tree.

Sorting and filtering with pstats

The pstats module gives you fine-grained control over how profiling data is displayed.

Sorting

sort_stats() accepts a string key. The most useful ones are:

• cumtime: cumulative time (the default recommendation).
• tottime: time in the function itself.
• calls: total call count.
• ncalls: primitive call count.
• name: alphabetical by function name.

You can chain sorts. stats.sort_stats("tottime", "cumtime") sorts by tottime first, then breaks ties with cumtime.

Filtering

print_stats() accepts a restriction argument to show only matching functions instead of dumping the entire table:

python

stats.print_stats("work")        # only functions containing "work"
stats.print_stats("my_module")   # only functions in my_module
stats.print_stats(0.1)           # only functions with at least 10% of total time

Filtering is useful when your program uses many library functions and you only care about your own code. Pass your module name or a function name prefix to narrow the output.

Limiting output

print_stats(N) shows only the first N rows, which prevents the full table from scrolling off screen. For large programs, 20 or 30 rows is usually enough to identify the main bottlenecks.

Saving profiles to a file

For programs that run for minutes or hours, you may want to save the profile and analyze it separately rather than printing it inline.

python

import cProfile
 
profiler = cProfile.Profile()
profiler.enable()
# ... program runs here ...
profiler.disable()
 
profiler.dump_stats("program.prof")

The .prof file can be loaded later, even in a different session or on a different machine, by pointing pstats.Stats at the saved path instead of a live profiler:

python

import pstats
 
stats = pstats.Stats("program.prof")
stats.sort_stats("cumtime")
stats.print_stats(20)

Saved profiles are also compatible with visualization tools like snakeviz and gprof2dot, which produce call graphs that make it easier to see which functions dominate the runtime at a glance. These tools are not in the standard library but are a pip install away when you need a visual overview.

Profiling only what matters

Profiling adds overhead. For most programs the overhead is small enough to ignore, but for programs that make millions of function calls per second, the profiler itself can slow things down noticeably.

When you only care about a specific section of code, enable and disable the profiler around that section rather than the whole program.

python

profiler = cProfile.Profile()
profiler.enable()
 
# Only this section is profiled.
result = expensive_operation(data)
 
profiler.disable()

The rest of the program runs at full speed, and you get profiling data only for the part you are investigating.

For even finer control, the profile decorator pattern lets you toggle profiling per function:

python

profiler = cProfile.Profile()
 
def profile_this(func):
    def wrapper(*args, **kwargs):
        profiler.enable()
        result = func(*args, **kwargs)
        profiler.disable()
        return result
    return wrapper
 
@profile_this
def suspect_function(data):
    return [x ** 2 for x in data if x % 2 == 0]

The decorator approach is lightweight and easy to add or remove as you narrow down the bottleneck.

A realistic profiling example

Consider a program that reads a JSON file of user records, filters by a condition, and writes a summary to a new file. Without profiling, you might guess that the JSON parsing is slow and spend time optimizing it.

python

import json
 
def load_records(path):
    with open(path) as f:
        return json.load(f)
 
def filter_active(records):
    return [r for r in records if r.get("status") == "active"]
 
def compute_summary(records):
    summary = {}
    for r in records:
        city = r.get("city", "unknown")
        summary[city] = summary.get(city, 0) + 1
    return summary
 
def main():
    records = load_records("users.json")
    active = filter_active(records)
    summary = compute_summary(active)
    return summary

These three functions look equally simple, so it is tempting to guess where the time goes. Profiling main() reveals which of the three steps actually dominates the runtime:

python

import cProfile
import pstats
import io
 
profiler = cProfile.Profile()
profiler.enable()
main()
profiler.disable()
 
stream = io.StringIO()
stats = pstats.Stats(profiler, stream=stream)
stats.sort_stats("cumtime")
stats.print_stats(10)
print(stream.getvalue())

The profile might reveal that load_records takes 80 percent of the runtime because the JSON file is large, while filter_active and compute_summary are fast by comparison. The right optimization is not to rewrite the filtering logic.

It might be to switch from json.load to ijson for streaming parsing, or to cache the parsed data, or to use a binary format for the data file.

The profile tells you where to focus, which is not always where you expect.

Common profiling pitfalls

Profiling with test-sized data

A profile run on a tiny test dataset may show different bottlenecks than one run on real data. A function that is fast with ten records might be slow with a million records because of algorithmic complexity. Always profile with input that represents real usage.

Profiling only once

System load, CPU frequency scaling, and disk cache state affect profiling results. Run the profile a few times and confirm the bottlenecks are consistent before acting on them.

Optimizing based on call count alone

A function called a million times with a nanosecond per-call cost may contribute less to runtime than a function called ten times with a second per-call cost. The tottime and cumtime columns should drive your decisions, not ncalls alone.

Forgetting to re-profile after changes

After you optimize a bottleneck, re-profile with the same input. Confirm that the function you changed now consumes less time and that no new bottleneck appeared. Optimization can shift the runtime profile in surprising ways.

When to use cProfile versus other tools

cProfile is the right starting point for most Python performance investigations. But it is not the only tool:

• Use timeit when comparing two small expressions in isolation.
• Use cProfile when you need to find the bottleneck in a program of any size.
• Use tracemalloc when memory usage, not speed, is the problem.
• Use a line profiler like line_profiler (third-party) when you have identified the slow function and need to know which line inside it is the culprit.

cProfile tells you which function is slow. A line profiler tells you which line in that function is slow. Together they form a complete investigative workflow: narrow from program to function to line.

The Python performance overview covers how these tools fit together. For specific optimization techniques once you have found the bottleneck, the article on optimizing Python loops is a practical next step.