Make HTTP Requests with Python `urllib`

The urllib package in the Python standard library lets you make HTTP requests without installing third-party libraries. It handles GET, POST, headers, redirects, and URL parsing. Use urllib when you need a built-in HTTP client and cannot or do not want to add the requests library as a dependency.

python

from urllib.request import urlopen
 
with urlopen("https://httpbin.org/get") as response:
    body = response.read().decode()
    print(response.status)
    print(body[:100])

The status prints as a plain integer, followed by the first 100 characters of the JSON body httpbin.org echoes back:

text

200
{
  "args": {},
  "headers": {
    "Accept-Encoding": "identity",
    "Host": "httpbin.org",

urlopen(url) sends a GET request and returns a response object. response.read() returns the full body as raw bytes, exactly as the server sent it. .decode() converts those bytes to a text string using the default encoding. Use with to ensure the underlying connection is closed once the block exits.

Making GET requests

GET requests fetch data from a URL, and query parameters need to be encoded correctly so special characters like spaces do not break the URL. Add query parameters with urllib.parse.urlencode().

python

from urllib.request import urlopen
from urllib.parse import urlencode
 
params = urlencode({"q": "python urllib", "page": "1"})
url = f"https://httpbin.org/get?{params}"
 
with urlopen(url, timeout=10) as response:
    print(response.status)                                    # 200
    print(response.headers.get("Content-Type"))  # application/json

urlencode() converts a dictionary to a properly escaped query string: q=python+urllib&page=1, handling spaces and other special characters automatically. Always set timeout to prevent hanging on unresponsive servers.

Making POST requests

Use urllib.request.Request with the data parameter for POST. Encode the data as bytes first, since urlopen expects the request body as raw bytes rather than a plain string.

python

from urllib.request import Request, urlopen
from urllib.parse import urlencode
 
data = urlencode({"username": "maya", "score": "92"}).encode()
request = Request("https://httpbin.org/post", data=data, method="POST")
 
with urlopen(request, timeout=10) as response:
    body = response.read().decode()
    print(response.status)   # 200

urlencode({"username": "maya"}) produces "username=maya&score=92", the standard application/x-www-form-urlencoded format most servers expect. .encode() converts that string to bytes, since urlopen requires the body as bytes rather than text. Request(..., data=data, method="POST") creates a POST request with the encoded form data in the body.

Sending JSON

To send JSON, serialize with json.dumps() and set the Content-Type header so the server knows how to parse the request body.

python

import json
from urllib.request import Request, urlopen
 
payload = json.dumps({"name": "Maya", "active": True}).encode()
headers = {"Content-Type": "application/json", "Accept": "application/json"}
 
request = Request("https://httpbin.org/post", data=payload, headers=headers, method="POST")
 
with urlopen(request, timeout=10) as response:
    result = json.loads(response.read())
    print(result["json"])   # {'name': 'Maya', 'active': True}

json.dumps() serializes the Python dict into a JSON string, which then gets encoded to bytes for the request body. The Content-Type header tells the server to expect and parse the body as JSON. The response in this case is also JSON, parsed back into a Python object with json.loads().

Setting custom headers

Pass a dictionary of headers to Request, or add them one at a time after the fact with add_header(), which is convenient when headers are built up conditionally.

python

from urllib.request import Request, urlopen
 
request = Request("https://httpbin.org/headers")
request.add_header("User-Agent", "MyPythonApp/1.0")
request.add_header("Authorization", "Bearer token123")
 
with urlopen(request, timeout=10) as response:
    print(response.read().decode())

Headers identify your client to the server, send authentication tokens like bearer or API keys, and control content negotiation such as which response format the server should return. Always set a User-Agent header; some APIs reject requests that arrive without one.

Handling errors

HTTP errors and network failures raise different exceptions, so catching them separately lets you respond differently to a bad request versus an unreachable server.

python

from urllib.request import urlopen
from urllib.error import HTTPError, URLError
 
try:
    with urlopen("https://httpbin.org/status/404", timeout=10) as response:
        print(response.read())
except HTTPError as e:
    print(f"HTTP error: {e.code} {e.reason}")
except URLError as e:
    print(f"Network error: {e.reason}")

Requesting the /status/404 endpoint triggers an HTTPError with both a numeric code and a text reason, which is caught before the more general URLError:

text

HTTP error: 404 NOT FOUND

HTTPError is raised for HTTP status codes 400 and above. URLError is raised for network-level problems like DNS failures or refused connections. HTTPError is a subclass of URLError, so catch HTTPError first.

Parsing URLs

urllib.parse provides functions for breaking a URL down into its components and for building new URLs from relative paths.

python

from urllib.parse import urlparse, urljoin
 
parsed = urlparse("https://example.com/path?query=value#section")
print(parsed.scheme)
print(parsed.netloc)
print(parsed.path)
print(parsed.query)
 
base = "https://example.com/docs/"
full = urljoin(base, "../api/v1/users")
print(full)

The first four lines break the URL into its parts; the last line shows urljoin() resolving ../api/v1/users one directory up from /docs/:

text

https
example.com
/path
query=value
https://example.com/api/v1/users

urlparse() splits a URL into its scheme, network location, path, query string, and fragment. urljoin() resolves relative URLs against a base URL, handling .. segments correctly the same way a browser would when following a relative link.

Practical example: fetching and parsing an API

Combine urllib with json to fetch and display API data. The function wraps the request in a try/except so a network failure prints a message instead of crashing.

python

import json
from urllib.request import urlopen
from urllib.error import URLError
 
def fetch_posts(limit=3):
    url = f"https://jsonplaceholder.typicode.com/posts?_limit={limit}"
    try:
        with urlopen(url, timeout=10) as response:
            posts = json.loads(response.read())
        for post in posts:
            print(f"{post['id']}: {post['title'][:50]}...")
    except URLError as e:
        print(f"Failed to fetch: {e.reason}")

Calling the function fetches the first three posts from the placeholder API and prints each post's id and a truncated title:

python

fetch_posts()

The default limit of 3 restricts the response to the first three posts, and each title is truncated to 50 characters for a compact display:

text

1: sunt aut facere repellat provident occaecati e...
2: qui est esse...
3: ea molestias quasi exercitationem repellat qui...

Common mistakes

Forgetting to set a timeout. Without timeout, a request to a dead or slow server hangs forever, and there is nothing built in to interrupt it. Always pass timeout=N to urlopen() so a network failure surfaces as an exception instead of a frozen program.

Not closing the response. Without with, the connection may stay open even after you are done reading the body. Always use with urlopen(...) as response: to ensure cleanup happens automatically, even if an exception occurs while reading the response.

Using urllib for complex HTTP workflows. urllib handles basic requests well but lacks sessions, automatic JSON handling, and connection pooling. For applications with many HTTP calls, authentication flows, or retry logic, use the requests library instead of reimplementing those features on top of urllib.

Encoding the URL manually instead of using urlencode. Hand-built query strings with + and & easily break with special characters like spaces, ampersands, or non-ASCII text in the values. Use urlencode() to handle encoding correctly every time.