Python Performance Optimization Best Practices

A practical checklist of Python performance best practices: what to measure, which tools to use, when to stop, and how to keep optimized code maintainable.

7 min read

This article is a practical checklist of Python performance best practices, not a tutorial. Each item is a principle you can apply in any Python project. All techniques are drawn from the earlier articles in this section.

The optimization hierarchy

Apply these in order. Each level gives larger gains than the one below it.

Level 1: Profile. Never optimize without data. Use timeit for micro-benchmarks, cProfile for whole programs, tracemalloc for memory. Find the single biggest bottleneck and fix only that.

Level 2: Algorithms and data structures. A set instead of a list for membership, a dict instead of a list of tuples for lookups, an O(n log n) sort instead of O(n squared) nested loops. These changes can deliver 100x speedups.

Level 3: Built-in functions and comprehensions. Functions like sum and max plus list comprehensions run in C. Replacing a manual loop with a built-in gives 3x to 10x speedups.

Level 4: Caching. If the same computation repeats with the same inputs, cache the result. functools.lru_cache is a one-line decorator with massive payoff for pure functions.

Level 5: Concurrency. When the bottleneck is waiting for I/O, overlap the waits with threading or asyncio. When the bottleneck is CPU, use multiprocessing or push work into NumPy.

Level 6: Micro-optimizations. Hoist invariants, cache attribute lookups, pre-allocate lists. These give 5 to 20 percent improvements. Do them last and only in confirmed hot loops.

The measurement toolkit

TaskTool
Compare two snippetstimeit
Find slowest functioncProfile
Track memory allocationstracemalloc
Check object sizesys.getsizeof
Line-level profilingline_profiler (pip install)

Use the tool that answers the question you have. Do not use cProfile for comparing two one-liners. Do not use timeit for finding the bottleneck in a 10,000-line program.

Keep it readable

Optimized code must still be understood by humans. A few rules keep your optimizations from becoming maintenance nightmares.

Name your variables so the intent is clear. A local variable called sqrt = math.sqrt is fine, but a local variable called s = math.sqrt is not.

The microsecond you save in typing is lost the first time someone wonders what s means.

Comment why, not what. The code already says what it does, so the comment should say why this approach was chosen.

For example: "Using set for O(1) membership; profiling showed list scan was 80 percent of runtime."

Do not chain optimizations. Make one optimization per change and measure separately.

This way you know which change helped and which did not. It also makes reverting easier if an optimization turns out to be harmful.

The stop condition

Define when you are done before you start. If the target is "API response under 200ms p99" and you reach 180ms, stop.

Do not spend two more days chasing 150ms. The user cannot tell the difference.

A performance target should be numeric, measurable, and tied to user experience. "The page should load faster" is not a target. "The page should load in under 2 seconds on a 4G connection" is.

Avoid these anti-patterns

Do not optimize code you have not profiled. Your intuition about what is slow is usually wrong.

Do not rewrite working code to be cleverer. A 5 percent speedup that makes the code unreadable is a net loss.

Do not optimize for a benchmark that does not represent real usage. A micro-benchmark showing a 40 percent improvement in a function called once at startup means nothing.

Do not remove error handling or input validation in the name of speed. Correctness comes first, always.

Do not cargo-cult optimizations you read online. What was fast in Python 3.6 may be irrelevant in Python 3.14, and what was fast on one workload may be slow on yours.

Measure on your code, with your data.

A final checklist

Before you close the performance ticket, confirm:

  • You profiled before and after and the numbers improved.
  • The code is still readable and the optimization is commented.
  • You did not break any tests.
  • The improvement actually matters to the user.
  • You are not tempted to optimize the next function without profiling it first.

For hands-on practice applying these best practices, the next article walks through optimizing a real Python project from profiling through fix. For the techniques behind each level of the hierarchy, the article on Python performance explained gives the full overview.

Rune AI

Rune AI

Key Insights

  • Always profile before optimizing. Never guess where the bottleneck is.
  • Fix the algorithm or data structure first. It yields the biggest gains.
  • Use built-in functions and comprehensions to push work into C.
  • Cache expensive computations that repeat with the same inputs.
  • Stop optimizing when the code meets its target. Readable code beats clever code.
RunePowered by Rune AI

Frequently Asked Questions

What is the single best Python performance practice?

Profile before you optimize. Every other practice depends on knowing where your program actually spends time. Without a profile, you are guessing. With a profile, you know exactly which function to fix and whether your fix helped.

Should I always use the fastest Python syntax?

No. Use the clearest syntax first. Only switch to a faster alternative when profiling shows that line is a bottleneck and the speedup actually matters. A list comprehension that is 20 percent faster but unreadable is not a best practice. A list comprehension that is 20 percent faster AND clear is a good default.

Conclusion

Python performance best practices form a hierarchy. Profile first. Fix algorithms and data structures before syntax. Move work into C-level built-ins. Cache expensive results. Add concurrency for I/O. Micro-optimize last. Above all, measure every change and stop when the code meets its performance target. Optimized code that nobody can read is not a best practice.