Why Tokens Exist

You already know that computers store text as numbers — every character maps to a code point, every code point to bytes. So when we feed text to a language model, the obvious question is: what unit should each number represent? A character? A word? Something in between? The answer to that question is the token, and it shapes everything about how modern AI models read and write.

The problem: what should one number represent?

Imagine you are designing a language model from scratch. Your model has a fixed vocabulary — a big lookup table where every entry gets an ID. Your first job is deciding what goes in that table.

Option 1: one character per entry

This is tempting. English needs maybe 100 entries (letters, digits, punctuation), and even all of Unicode is a bounded set. Nothing is ever "unknown" — any text can be spelled out.

But there's a cost. The sentence The cat sat on the mat becomes 23 separate steps. Models pay for every step: attention cost grows quickly with sequence length, and the model has to learn that c, a, t in a row means a furry animal before it can learn anything about cats. You are spending most of your model's capacity re-learning spelling.

Option 2: one word per entry

Great, now cat is a single ID and the sentence is 6 steps. But English has hundreds of thousands of words — and that's before you count cats, catlike, caterpillar, typos, names, hashtags, and other languages. Your table explodes into the millions and is still incomplete: the first time someone types Skibidi or a new product name, your model has literally no ID for it. Word-level models need a special UNK (unknown) token, which is like reading a book where random words are blacked out.

The middle ground: subwords

The fix is elegant: let common strings be single entries, and let rare strings be built from pieces.

• the is common → one token.
• cat is common → one token.
• caterpillar is rarer → maybe cat + erp + illar.
• Skibidi is unknown → spelled from smaller chunks, but never impossible.

The most popular way to build such a vocabulary is Byte Pair Encoding (BPE). The intuition is beautifully simple: it's just greedy compression.

The result: frequent words get one ID, rare words get a few IDs, and nothing is ever unknown, because in the worst case we fall back to raw bytes.

See it yourself

Type into the visualizer below. Notice how common English words become single colored chunks, while unusual words shatter into pieces. Try cat, then caterpillar, then something you invent.

A tiny BPE in code

Here is the core merge loop, stripped to its essence:

from collections import Counter

def most_common_pair(tokens):
    pairs = Counter(zip(tokens, tokens[1:]))
    return pairs.most_common(1)[0][0]

def merge(tokens, pair, new_id):
    out, i = [], 0
    while i < len(tokens):
        if i < len(tokens) - 1 and (tokens[i], tokens[i+1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(tokens[i])
            i += 1
    return out

text = "the theme of the thesis"
tokens = list(text.encode("utf-8"))  # start from raw bytes
for step in range(10):
    pair = most_common_pair(tokens)
    tokens = merge(tokens, pair, 256 + step)
print(tokens)  # shorter every iteration

Real tokenizers add details (byte-level tricks, special tokens, pre-splitting on spaces), but this loop is genuinely the heart of BPE.

Build it yourself

1. Run the code above on a paragraph of your own writing. Print the sequence length after each merge — watch it shrink.
2. In the visualizer, find a word that splits into exactly 3 tokens. Can you explain why it split where it did?
3. Predict before you type: will internationalization be more or fewer tokens than xzqjvw? Check your guess.

Summary

• Models read integers, so text must be chopped into units with IDs — the design question is what one unit represents.
• Character-level: tiny vocabulary, but painfully long sequences.
• Word-level: short sequences, but a huge and forever-incomplete vocabulary.
• Subword tokens (BPE) are the compromise: common strings become single tokens, rare strings decompose into pieces, and byte fallback means nothing is ever unknown.
• Rule of thumb for English: about 4 characters or three-quarters of a word per token.