Unicode and Encoding in Python

Unicode and encoding in Python are the concepts that explain how text moves between the inside of your program, where every string is a clean sequence of Unicode characters, and the outside world, where text lives in files, travels over networks, and sits in databases as raw bytes. Every time you read a text file, fetch a web page, parse a JSON response, or store user input in a database, Python converts between its internal Unicode representation and an external byte representation using a specific encoding. Understanding this conversion is essential because encoding errors are among the most common issues that beginners encounter when working with real data from real sources.

Inside your Python program, every string is already Unicode. You do not need to do anything special to work with text in any language, and characters from different scripts coexist naturally in the same string. The article on Python strings introduced the concept of strings as sequences of Unicode code points. This article focuses on what happens at the boundary where strings meet bytes, which is where encoding and decoding happen. For working with multiline text and file contents, the article on multiline strings covers related concepts.

Python 3 made a clean separation between text and binary data that Python 2 did not have. The str type is always Unicode text, and the bytes type is always raw binary data. You cannot accidentally mix them, and any attempt to combine a string with bytes raises a TypeError. This separation forces you to think explicitly about encoding at the boundaries of your program, which prevents an entire category of subtle bugs that plagued Python 2 code.

How encode() converts strings to bytes

The encode() method is called on a string and returns a bytes object that represents the same text in a specific byte encoding. The most important encoding is UTF-8, which is the default in Python and the dominant encoding across the internet. UTF-8 can represent every character in the Unicode standard, it is backward-compatible with ASCII for characters in the 0 through 127 range, and it uses a variable number of bytes per character, which makes it efficient for text that is primarily Latin script while still supporting every other script in the world.

python

text = "Python and Unicode: café and 日本語"
encoded = text.encode("utf-8")
print(type(encoded))  # <class 'bytes'>
print(encoded)        # b'Python and Unicode: caf\xc3\xa9 and \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e'

The output shows that the ASCII characters like P, y, t, h, o, n appear as their literal byte values inside the bytes representation, while non-ASCII characters like the accented e in café and the Japanese characters appear as hexadecimal escape sequences. Each escape represents one or more bytes that UTF-8 uses to encode that character. The string café takes five bytes in UTF-8 because the four ASCII letters use one byte each and the accented e uses two bytes.

You can specify other encodings if your use case requires them. Latin-1 encodes Western European characters in a single byte each, ASCII strictly limits to the first 128 Unicode code points, and UTF-16 uses at least two bytes per character. In almost all modern applications, UTF-8 is the correct choice, and you should only use another encoding when you are interoperating with a legacy system that specifically requires it.

How decode() converts bytes back to strings

The decode() method reverses the encoding process. It is called on a bytes object and returns a string, interpreting the bytes according to the encoding you specify. The encoding you pass to decode() must match the encoding that was used to create the bytes in the first place. If you decode with the wrong encoding, Python may produce garbled text or raise a UnicodeDecodeError.

The most common source of bytes that need decoding is reading from a file. When you open a file in text mode, Python handles decoding automatically using the encoding you specify or the platform default. When you open a file in binary mode, you receive raw bytes and must call decode() yourself. The same pattern applies to network data, database results, and any other external source of bytes.

python

byte_data = b"Python and Unicode: caf\xc3\xa9 and \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e"
decoded = byte_data.decode("utf-8")
print(decoded)  # Python and Unicode: café and 日本語

Encoding errors happen when the bytes contain sequences that are not valid in the specified encoding. The default error handling mode, called strict, raises a UnicodeDecodeError. You can change this behavior by passing an errors argument. The value ignore silently skips problematic bytes, replace substitutes the Unicode replacement character, and backslashreplace produces escape sequences that show the problematic bytes, which is useful for debugging. In production code, strict mode is usually correct because silently corrupting data is worse than failing loudly with a clear error message.