Strings and Bytes

Last lesson ended on a cliffhanger: a code point is an abstract number, not what actually sits in memory. This lesson closes the loop — how strings become bytes (encoding), how bytes become strings again (decoding), why UTF-8 won the encoding wars, and what happens when this handshake goes wrong (hello, mojibake).

The problem

Here's the naive solution: since the biggest code points need about 21 bits, just use 4 bytes for every character. This exists (it's called UTF-32) and it works — but it makes every English document four times larger than ASCII, since a plain letter like A wastes three zero bytes. The internet was not going to accept 4x bandwidth for nothing.

UTF-8's insight: use a variable number of bytes. Common characters get short encodings; rare ones get longer ones.

UTF-8: the variable-width trick

The byte patterns are self-describing: the first byte's leading bits announce how many bytes the character uses, and continuation bytes have a distinctive signature. A decoder dropped into the middle of a stream can always find the next character boundary — a genuinely elegant design.

Encode and decode

Encoding turns a string into bytes; decoding turns bytes back into a string. Both require naming the scheme — and both sides must name the same scheme.

s = "café 🎉"

data = s.encode("utf-8")
print(data)
# b'caf\xc3\xa9 \xf0\x9f\x8e\x89'

print(len(s))      # 6  characters (code points)
print(len(data))   # 10 bytes: c,a,f = 3, é = 2, space = 1, 🎉 = 4

print(data.decode("utf-8"))   # 'café 🎉'  — round trip complete

The same in JavaScript:

const bytes = new TextEncoder().encode("café 🎉");
console.log(bytes.length);                    // 10
console.log(new TextDecoder().decode(bytes)); // 'café 🎉'

Mojibake: when the handshake fails

Encode with one scheme, decode with another, and you get mojibake (from Japanese: "character transformation") — the classic garbled text:

data = "café".encode("utf-8")     # é becomes bytes 195, 169
print(data.decode("latin-1"))     # 'café'   — oops

What happened: in UTF-8, é is the two-byte sequence 195, 169. A Latin-1 decoder doesn't know about multi-byte sequences — it maps each byte to its own character: 195 is à and 169 is ©. Whenever you've seen ’ where an apostrophe should be, or é in an email, you've witnessed exactly this bug. The data isn't corrupted — it's being read with the wrong table, and re-decoding with the right one recovers it.

Why AI engineers care

This is the exact floor that LLMs stand on:

• Tokenizers eat bytes. Modern tokenizers (like the byte-pair encoding used by GPT-style models) start from the UTF-8 bytes of your text. This is why an emoji can cost multiple tokens while the word "the" costs one, and why non-English text is often more expensive per sentence — more bytes in, more tokens out.
• Context windows and bills are downstream of bytes. Understanding characters vs bytes vs tokens is the difference between guessing and knowing when you budget a context window.
• APIs move bytes. Every request you'll send to a model provider is UTF-8 encoded JSON over the wire. When something garbles, you now know exactly where to look.

Build it yourself

Write a byte inspector, then deliberately cause and repair mojibake:

def inspect(s):
    data = s.encode("utf-8")
    print(f"text: {s!r}")
    print(f"code points: {len(s)}, bytes: {len(data)}")
    for ch in s:
        b = ch.encode("utf-8")
        print(f"  {ch!r}  U+{ord(ch):04X}  ->  {list(b)}  ({len(b)} bytes)")

inspect("Go 中 🎉")

Then the repair drill: take garbled = "café".encode("utf-8").decode("latin-1") — print it to admire the damage — and recover the original with garbled.encode("latin-1").decode("utf-8"). Once you've un-mojibaked a string on purpose, you'll never be mystified by é in the wild again.

Summary

• An encoding is the packing scheme between abstract code points and physical bytes; UTF-8 is the universal standard.
• UTF-8 is variable-width: 1 byte for ASCII, up to 4 for emoji — efficient and perfectly backwards compatible.
• Characters, bytes, and (in JavaScript) code units are three different length measurements for the same string; know which one your API means.
• Mojibake = encoded with one scheme, decoded with another; the data is recoverable, the fix is agreeing on UTF-8.
• LLM tokenizers sit directly on UTF-8 bytes — this lesson is literally the input layer of every model you'll use. Next stop: tokenization.