Build an Asynchronous Web Scraper with Python

A web scraper that fetches 200 pages one at a time is a test of patience. At 200 milliseconds per page, the synchronous version takes 40 seconds. An async version using asyncio and aiohttp fetches all 200 pages concurrently and finishes in roughly 200 milliseconds plus scheduling overhead, roughly a 200x speedup.

This article builds a complete async web scraper from scratch, combining the async patterns covered throughout this section into a working project. If you have read about running multiple tasks concurrently and async context managers, you already know the building blocks: gathering coroutines, limiting concurrency with a semaphore, and managing a session as a context manager. This article assembles them into a real application rather than introducing anything new.

The scraper fetches a list of URLs concurrently, limits concurrency to avoid overwhelming the target server, retries on transient failures, parses HTML to extract data, and writes results to a CSV file. Every component is an async pattern you can reuse in other projects, whether you are scraping a different site entirely or swapping the output format from CSV to a database or JSON file. Read through each step in order, since later steps build directly on the functions defined in earlier ones.

Project overview

The scraper has five components, and each one is built and explained as its own step before they are combined into the final program:

1. URL generation: produce the list of pages to scrape.

2. Concurrent fetching: fetch all pages with aiohttp, limited by a semaphore.

3. Error handling: retry transient failures with exponential backoff.

4. HTML parsing: extract data with BeautifulSoup.

5. Output: write results to a CSV file.

The complete program is about 80 lines of code. Each component is independently testable and can be replaced without changing the others, so swapping BeautifulSoup for a different parser or CSV for a database only touches one function.

Installation

Install the required libraries:

text

pip install aiohttp beautifulsoup4

aiohttp provides the async HTTP client that does the actual concurrent fetching. beautifulsoup4 parses the HTML each response contains into something you can query for specific elements. Both are mature, well-documented libraries as of July 2026, with large communities and stable APIs.

Step 1: Fetch a single page

Start with the simplest async fetch and build from there:

python

import asyncio
import aiohttp
 
async def fetch(session, url):
    async with session.get(url, timeout=10) as response:
        response.raise_for_status()
        return await response.text()
 
async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, "https://httpbin.org/html")
        print(f"Fetched {len(html)} characters")
 
asyncio.run(main())

ClientSession reuses TCP connections across requests, which is significantly faster than opening a new connection for every fetch. Create one session and pass it to every fetch call rather than creating a session per request. The timeout parameter prevents hanging on unresponsive servers that accept a connection but never send a response.

raise_for_status() raises an exception on HTTP errors like 404 or 500, which is what lets the retry logic added in a later step detect and react to a failed request instead of silently treating an error page as valid content.

Step 2: Add concurrency with semaphore limiting

Fetching one page is sequential. To fetch many, create tasks with asyncio.gather(). To avoid overwhelming the server, use a semaphore to limit concurrent connections:

python

import asyncio
import aiohttp
 
async def fetch(session, sem, url):
    async with sem:
        async with session.get(url, timeout=10) as response:
            response.raise_for_status()
            return await response.text()
 
async def fetch_all(urls, concurrency=10):
    sem = asyncio.Semaphore(concurrency)
 
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, sem, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

Running this against 20 URLs with a concurrency of 5 shows how many fetches succeeded versus failed, without letting a single failure abort the whole batch:

python

async def main():
    urls = [f"https://httpbin.org/delay/0.1" for _ in range(20)]
    results = await fetch_all(urls, concurrency=5)
 
    successes = sum(1 for r in results if not isinstance(r, Exception))
    errors = sum(1 for r in results if isinstance(r, Exception))
    print(f"Fetched {successes} pages, {errors} errors")
 
asyncio.run(main())

At most 5 connections run concurrently, no matter how many URLs are in the list. When the sixth task reaches async with sem, it blocks until one of the first five completes and releases the semaphore, and the next waiting task takes its place. This pattern is documented in detail in the article on running multiple tasks concurrently.

return_exceptions=True prevents a single failed fetch from cancelling all other fetches. Without it, gather() would raise the first exception it sees and cancel every other in-flight task, turning one bad URL into a total scraping failure. Instead, exceptions appear in the results list as Exception objects rather than being raised, so the caller can decide what to do with each failed URL individually.

Step 3: Add retry with exponential backoff

Network requests fail. Servers return transient 503 errors. Connections time out.

A robust scraper retries failed requests with increasing delays:

python

import asyncio
import aiohttp
import random
 
async def fetch_with_retry(session, sem, url, max_retries=3):
    for attempt in range(max_retries):
        try:
            async with sem:
                async with session.get(url, timeout=10) as response:
                    if response.status == 429:
                        retry_after = int(response.headers.get("Retry-After", 5))
                        await asyncio.sleep(retry_after)
                        continue
                    response.raise_for_status()
                    return await response.text()
        except (aiohttp.ClientError, asyncio.TimeoutError) as e:
            if attempt == max_retries - 1:
                raise
            delay = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(delay)
    raise RuntimeError(f"Max retries exceeded for {url}")

The backoff formula 2 ** attempt produces delays of roughly 1, 2, 4 seconds across three attempts, growing each time so a persistently failing request backs off further rather than hammering the server at a fixed rate. The random jitter prevents multiple retries from synchronizing and hitting the server simultaneously, which matters when many concurrent requests fail around the same time and would otherwise all retry in lockstep. If the server returns a 429 (Too Many Requests) with a Retry-After header, the scraper respects that value instead of using its own backoff, since the server is telling you exactly how long to wait.

Step 4: Parse HTML with BeautifulSoup

With the HTML in hand, extract data using BeautifulSoup. This example extracts the page title and all paragraph text:

python

from bs4 import BeautifulSoup
 
def parse_page(html):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.string if soup.title else "No title"
    paragraphs = [p.get_text(strip=True) for p in soup.find_all("p")]
    return {"title": title, "paragraph_count": len(paragraphs)}

The previous example grabs every paragraph on the page, which works for generic content but not for structured pages like product listings. Use CSS selectors with soup.select() or soup.select_one() for more targeted extraction of specific elements by class or tag:

python

def parse_product_page(html):
    soup = BeautifulSoup(html, "html.parser")
    name = soup.select_one(".product-name")
    price = soup.select_one(".product-price")
    return {
        "name": name.get_text(strip=True) if name else None,
        "price": price.get_text(strip=True) if price else None,
    }

The if name else None fallback matters because select_one() returns None when nothing on the page matches the selector, and calling get_text() on None would raise an AttributeError instead of just leaving that field empty for pages with a slightly different layout.

Step 5: Write results to CSV

Use Python's csv module to write results as they are parsed. The csv module is synchronous, so for large datasets consider writing in batches or using aiofiles for fully async I/O:

python

import csv
 
def save_results(results, filename="scraped_data.csv"):
    if not results:
        return
 
    fieldnames = results[0].keys()
 
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(results)
 
    print(f"Saved {len(results)} rows to {filename}")

csv.DictWriter reads its column order from fieldnames, so it writes a consistent header row even though the dictionaries it receives come from parsing HTML that can vary from page to page.

Complete scraper

The five components above each solve one piece of the problem in isolation. Combining them into a single program shows how fetching, retrying, parsing, and saving fit together as one working scraper:

python

import asyncio
import aiohttp
import csv
import random
from bs4 import BeautifulSoup
 
async def fetch_page(session, sem, url, max_retries=3):
    for attempt in range(max_retries):
        try:
            async with sem:
                async with session.get(url, timeout=10) as response:
                    if response.status == 429:
                        retry_after = int(response.headers.get("Retry-After", 5))
                        await asyncio.sleep(retry_after)
                        continue
                    response.raise_for_status()
                    return await response.text()
        except (aiohttp.ClientError, asyncio.TimeoutError) as e:
            if attempt == max_retries - 1:
                return e
            delay = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(delay)
    return RuntimeError(f"Max retries exceeded for {url}")

fetch_page returns the exception object instead of raising it, so one failed URL does not stop asyncio.gather() from collecting the rest. parse_page checks for that exception and turns it into an error row instead of trying to parse it as HTML:

python

def parse_page(html, url):
    if isinstance(html, Exception):
        return {"url": url, "error": str(html)}
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.string if soup.title else ""
    text = soup.get_text(separator=" ", strip=True)[:200]
    return {"url": url, "title": title, "preview": text}
 
async def scrape_urls(urls, concurrency=10):
    sem = asyncio.Semaphore(concurrency)
 
    async with aiohttp.ClientSession(
        headers={"User-Agent": "Python-Async-Scraper/1.0"}
    ) as session:
        tasks = [fetch_page(session, sem, url) for url in urls]
        html_results = await asyncio.gather(*tasks)
 
    return [parse_page(html, url) for html, url in zip(html_results, urls)]

With fetching and parsing in place, the remaining pieces write results to disk and tie everything together into a runnable program:

python

def save_csv(results, filename="scraped_data.csv"):
    fieldnames = ["url", "title", "preview", "error"]
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames,
                                extrasaction="ignore")
        writer.writeheader()
        writer.writerows(results)
 
async def main():
    urls = [
        "https://httpbin.org/html",
        "https://httpbin.org/links/10/0",
    ]
 
    results = await scrape_urls(urls, concurrency=3)
    save_csv(results)
 
    for row in results:
        if "error" in row:
            print(f"ERROR: {row['url']} - {row['error']}")
        else:
            print(f"OK: {row['url']} - {row['title']}")
 
asyncio.run(main())

Both test URLs fetch successfully here, so every row prints as OK and the CSV file ends up with one row per URL instead of any error rows:

text

OK: https://httpbin.org/html - Herman Melville - Moby-Dick
OK: https://httpbin.org/links/10/0 - Links powered by httpbin.org
Saved 2 rows to scraped_data.csv

Adapting the scraper for real websites

The scraper above works with any website. Adapt the parse_page function to match the HTML structure of your target:

python

def parse_product_page(html, url):
    if isinstance(html, Exception):
        return {"url": url, "error": str(html)}
    soup = BeautifulSoup(html, "html.parser")
    return {
        "url": url,
        "name": soup.select_one("h1.product-title").get_text(strip=True),
        "price": soup.select_one("span.price").get_text(strip=True),
        "description": soup.select_one("div.description").get_text(strip=True)[:500],
    }

This version keeps the same isinstance(html, Exception) check as parse_page, but swaps the generic paragraph extraction for CSS selectors that target the exact elements a product page uses, so you only need to change this one function to point the scraper at a new site's layout.

Many target sites spread results across dozens of numbered pages rather than a single URL. For sites with pagination, generate the full list of URLs in a loop before passing it to scrape_urls:

python

def generate_urls(base_url, total_pages):
    return [f"{base_url}?page={p}" for p in range(1, total_pages + 1)]

The exact query parameter and page numbering scheme varies by site, so check the site's own pagination links to confirm the pattern before generating hundreds of URLs from a guess.

Some pages are only visible to logged-in users. For sites that require authentication, add the appropriate headers or cookies to the ClientSession so every request it makes carries them automatically:

python

session = aiohttp.ClientSession(
    headers={
        "User-Agent": "Python-Async-Scraper/1.0",
        "Authorization": "Bearer YOUR_TOKEN",
    },
    cookies={"session_id": "abc123"},
)

Headers and cookies set on the ClientSession apply to every request made through it, so you configure authentication once instead of repeating it on each individual session.get() call.

Responsible scraping guidelines

The speed of an async scraper makes it easy to accidentally overwhelm a target server. Follow these guidelines:

• Check robots.txt before scraping. Parse it and exclude disallowed paths.
• Set a descriptive User-Agent including contact information so site operators can reach you.
• Limit concurrency to a reasonable number, typically 5 to 10 concurrent requests for small sites, more for large CDN-backed sites.
• Respect 429 responses and Retry-After headers. Slow down when the server asks you to.
• Cache responses locally during development so you do not repeatedly hit the same server while debugging your parser.
• Run during off-peak hours if scraping a significant number of pages.

A scraper built with these practices is a legitimate tool for data collection. One that ignores them is a denial-of-service attack, even if that was never the intent, because a fast async client with no limits can generate far more load than the person running it realizes. The performance tips in the previous article on concurrent Python performance cover additional techniques for making your scraper efficient and well-behaved.