Temperature, Top-p, and Sampling

You know that an LLM outputs a probability distribution over its whole vocabulary at every step. But a distribution isn't text — something has to pick a token. That picking step is called sampling, and the dials that control it — temperature and top-p — are the ones you'll actually turn every day as an AI engineer. This lesson makes them mechanical instead of magical.

Turn the dials yourself

Below is a fixed next-token distribution for "The weather today is ___". Move the sliders, watch the bars reshape, and hit Sample a bunch of times at different settings.

Experiments to run:

1. Temperature → 0.05. The distribution collapses onto "sunny". Sample ten times — you get "sunny" ten times. This is (near-)greedy decoding.
2. Temperature → 2. The bars flatten. Suddenly "apocalyptic" is in play. Sample repeatedly and enjoy the chaos.
3. Top-p → 0.5. Watch the tail get struck through: only the few tokens covering 50% of probability mass survive, no matter what temperature does.

Temperature: reshaping the distribution

Temperature is applied before the softmax that turns the model's raw scores (logits) into probabilities. Every logit is divided by T:

def softmax_with_temperature(logits, T):
    scaled = [l / T for l in logits]          # T < 1 exaggerates gaps
    exps = [math.exp(s) for s in scaled]      # T > 1 shrinks gaps
    total = sum(exps)
    return [e / total for e in exps]

• T < 1: dividing by a small number stretches the differences between logits apart; after the exponential, the top token dominates. Output becomes focused and repeatable.
• T = 1: the model's learned distribution, untouched.
• T > 1: differences shrink, the distribution flattens, low-probability tokens get real chances. Output becomes diverse — and error-prone.

Important mental correction: temperature doesn't make the model "smarter" or "dumber", and it doesn't add knowledge. It only reshapes how the existing distribution is sampled.

Top-p: cutting off the long tail

Even at moderate temperature, the vocabulary's long tail (tens of thousands of barely-possible tokens) collectively holds meaningful probability, and once in a while sampling lands there — producing that one bizarre word that ruins a paragraph.

Top-p (nucleus) sampling fixes this: sort tokens by probability, keep the smallest set whose cumulative probability reaches p (say 0.9), throw away everything else, renormalize, sample. The clever part: the kept set's size adapts. When the model is confident, the nucleus might be 2 tokens; when many continuations are plausible, it might be 50.

Settings cheat sheet

| Task | Temperature | Top-p | Why | |---|---|---|---| | Data extraction, classification | 0–0.2 | 1.0 | You want the same right answer every time | | Code generation | 0–0.4 | 1.0 | Syntax has little room for creativity | | General chat / assistants | 0.6–0.8 | 0.9–1.0 | Natural variety without going off the rails | | Brainstorming, fiction | 0.9–1.3 | 0.95 | Deliberately explore unlikely continuations |

One caveat worth internalizing: temperature 0 does not guarantee correctness — it gives you the model's most probable answer, deterministically. If the model's best guess is wrong, you'll get the same wrong answer very reliably.

Summary

• Sampling turns the model's distribution into an actual token; the strategy dramatically shapes output character.
• Temperature divides logits before softmax: low = sharp and deterministic, high = flat and adventurous.
• Top-p keeps only the adaptive "nucleus" of tokens covering p of the probability mass, chopping the risky tail.
• Match settings to the task: near-0 for extraction and code, ~0.7 for chat, higher for creative work.
• Temperature changes randomness, never knowledge — T = 0 is consistent, not correct.