distiliaa Learn
Module 5 15 min

Next-Token Prediction

The single idea behind every LLM: guess what comes next.

Everything you've learned so far — tokens, embeddings, latent representations — was the supporting cast. Now the star: the large language model. And here's the anticlimax that makes it beautiful: under all the hype, an LLM does exactly one thing. Given some text, it predicts what token comes next. That's it. That's the whole job.

Why does this exist?

Language understanding resisted decades of hand-written rules — grammar is too messy, meaning too contextual. Next-token prediction sidestepped the problem: instead of teaching a machine language, let it read trillions of tokens and learn to guess what comes next. It turns out that to guess well, you're forced to absorb grammar, facts, reasoning patterns, and style. One simple objective, learned at massive scale, produced the most capable AI systems ever built.

Play the game the model plays

Before any theory, feel the task yourself. Below is exactly what an LLM does billions of times during training: see a context, bet on the next word. Play a few rounds and compare your intuition to the "model's" probabilities.

Round 1 / 5You 0.00 · Model 0.00

Predict the next word: “The cat sat on the ___”

Notice what you just did to score well. For "The cat sat on the ___" you used world knowledge (cats like mats and sofas, not ceilings). For "To be or not to ___" you used memorized text. For "import numpy as ___" you used coding conventions. Predicting the next token well requires knowing an enormous amount about the world — which is exactly why training on this task produces knowledgeable models.

It's a probability distribution, not a guess

A crucial detail: the model doesn't output one word. It outputs a probability for every token in its vocabulary — often 100,000+ numbers that sum to 1.

Context: "The cat sat on the"

mat       ██████████████████░░  55%
sofa      ████████░░░░░░░░░░░░  25%
keyboard  █████░░░░░░░░░░░░░░░  15%
ceiling   █░░░░░░░░░░░░░░░░░░░   5%
...       (100,000 more tokens, each with a tiny probability)

Generation is a loop: predict the distribution, pick a token from it, append it to the context, repeat.

  1. Tokenize the promptYour text becomes a sequence of token IDs (Module 2!).
  2. Predict a distribution
  3. Pick one token
  4. Append and repeat

This loop is called autoregressive generation, and it explains many things you've observed: why responses stream word by word, why long outputs take longer, and why an early wrong token can send the whole answer off the rails (the mistake becomes context for everything after it).

How does it learn? By grading its guesses

Training is conceptually simple:

for text in trillions_of_tokens:
    context, actual_next = text[:i], text[i]
    predicted_probs = model(context)
    # loss is high when the model gave the real next token low probability
    loss = -log(predicted_probs[actual_next])
    loss.backward()   # nudge billions of weights to do better

No labels, no human grading — the text itself provides the answer, just like the autoencoder's input provided its own target. This is self-supervised learning at planetary scale: the internet is the textbook and the answer key.

'It's just autocomplete' — true and misleading

Yes, an LLM is autocomplete. But to autocomplete "The proof follows from Fermat's little theorem because…" you need to know number theory. To autocomplete a patient note you need medical knowledge. Compressing the ability to continue any text forces a model to internalize much of what humans have written down. Simple objective, profound consequences.

What this explains about LLM behavior

Keep the next-token frame in your pocket — it demystifies almost everything:

  • Confident nonsense: the model always outputs a plausible-sounding next token, whether or not it "knows" the fact (the hallucinations lesson digs in).
  • Prompt sensitivity: change the context, change the distribution. Prompting is literally steering probabilities.
  • No inner goals: the model isn't "trying" to do anything but continue text. Chat behavior comes from training on text formatted as helpful conversations.

Build it yourself

Build a tiny character-level next-token predictor: count, for each character in some text (say, a book from Project Gutenberg), the frequency of the character that follows it. Then generate text by repeatedly sampling from those counts. It will produce gibberish with eerily English-like rhythm — and you'll have built a (very small) language model with the exact same objective as GPT-4.

Summary

  • An LLM does one thing: given context, output a probability distribution over the next token.
  • Generation is a loop — sample a token, append, repeat (autoregressive).
  • Training is self-supervised: real text supplies both the question and the answer, at trillion-token scale.
  • Predicting well requires absorbing grammar, facts, and reasoning — that's why the simple objective yields capable models.
  • This frame explains streaming output, prompt sensitivity, and (soon) hallucinations and sampling dials.