The Interactive Tokenizer

In the last lesson you learned why tokens exist. Now we get our hands dirty. This whole lesson revolves around one tool — the interactive tokenizer below — and a series of experiments that will permanently change how you look at text going into a model.

The problem: your mental model of text is wrong

You see Hello, world! as 13 characters or 2 words. The model sees something else entirely — maybe 4 tokens with IDs like 9906, 11, 1917, 0. Neither characters nor words survive the trip. Until you internalize what the model actually receives, you will keep making wrong predictions about length, cost, and behavior.

Experiment 1: tokens and their IDs

Type Hello, world! into the visualizer and look at two things: the colored chunks (the tokens) and the numbers underneath (the IDs).

Every token is just a row number in the tokenizer's vocabulary table. The model never sees letters — it sees 9906 and has learned, from training data, what tends to follow 9906. Two consequences worth pausing on:

• The same word can map to different IDs depending on context (more on spaces below).
• The model has no built-in access to the letters inside a token. This is why models are famously bad at questions like counting the letter r in strawberry — the word may arrive as one or two opaque chunks, not as ten letters.

Experiment 2: the mystery of the leading space

Type hello and then hello (with a leading space). Different tokens, different IDs.

Most modern tokenizers attach the space to the front of the following word, so the vocabulary contains both hello and ␣hello as separate entries. Why? Because in real text, words almost always follow a space. Baking the space into the token means The cat sat is 3 tokens (The, ␣cat, ␣sat) instead of 5 (The, ␣, cat, ␣, sat). It's pure compression.

Experiment 3: capitalization and word position

Try these four, one at a time: dog, Dog, DOG, dOg.

You'll likely see dog and Dog as single (different!) tokens, while DOG and dOg may split. The tokenizer merged whatever was frequent in training text — lowercase and capitalized-first-letter forms are common; SHOUTING and tYpOs are not. To the model, dog and Dog are as unrelated at the input layer as dog and banana; any connection between them had to be learned.

Experiment 4: numbers, code, and other languages

Now the weird cases. Try each of these and watch the token count:

1. Numbers. Type 1234567890. Long numbers get chopped into arbitrary chunks like 123, 456, 7890. The splits do not respect place value — one reason arithmetic is hard for language models. Compare 42 (one token) against 424242424242.
2. Code. Paste a few lines of Python. Notice that indentation, def, and common operators tokenize efficiently — modern tokenizers were trained on lots of code and even have dedicated tokens for runs of spaces. Older tokenizers spent one token per space of indentation, making code hilariously expensive.
3. Non-English text. Type hello, then its Hindi translation नमस्ते, then some Chinese like 你好世界. The English is 1 token; the others may be 3-10 tokens for the same meaning. Vocabularies are trained mostly on English-heavy data, so other scripts get fewer merges and fragment into near-bytes. Same sentence, several times the cost and context usage.

Counting tokens in code

import tiktoken  # OpenAI's tokenizer library

enc = tiktoken.get_encoding("cl100k_base")

for text in ["hello", " hello", "नमस्ते", "1234567890"]:
    ids = enc.encode(text)
    print(f"{text!r:>15} -> {len(ids)} tokens: {ids}")

# Round-trip: IDs back to text
print(enc.decode(enc.encode("Tokens are reversible")))

Every serious LLM app ends up with a function like count_tokens(text) somewhere — for truncating documents, budgeting context, or estimating cost before sending a request.

Build it yourself

1. The compression game. Find the shortest string (in characters) that produces 5 or more tokens. Then find the longest string that stays at 1 token.
2. Language tax audit. Take one sentence, translate it into two other languages you know (or use a translator), and record the token count for each. Compute the "multiplier" versus English.
3. Boundary hunt. Find a case where adding a single character to the end of a string changes the tokens at the beginning. (Hint: try building up a long word letter by letter.)

Summary

• Models see token IDs — vocabulary row numbers — never letters, which explains failures like letter-counting.
• Leading spaces are folded into tokens as a compression trick; hello and ␣hello are different tokens, and this causes real prompt-engineering bugs.
• Case matters: dog, Dog, and DOG are unrelated entries at the input layer.
• Numbers split arbitrarily, code tokenizes well on modern tokenizers, and non-English text can cost several times more tokens for the same meaning.
• Token counts are tokenizer-specific — always count with the tokenizer of the model you actually call.