Attention — distiliaa Learn

If next-token prediction is what an LLM does, attention is how it does it well. Attention is the mechanism that lets a model, while processing one word, look back at every other word and decide which ones matter right now. It's the "T" in GPT — the Transformer is built almost entirely out of attention layers.

Why does this exist?

Before 2017, models read text like a person with severe short-term memory loss: word by word, squeezing everything seen so far into one fixed-size summary. Long sentences overwhelmed that summary — by the end, the beginning was mush. Attention fixed this by giving the model direct access to every previous word at every step, with learned weights deciding relevance. The 2017 paper "Attention Is All You Need" showed you could build the whole model from this one mechanism — and the Transformer it introduced powers every modern LLM.

The problem attention solves

Read this sentence:

"The animal didn't cross the street because it was too tired."

What does "it" refer to? The animal, obviously — streets don't get tired. Now change one word:

"The animal didn't cross the street because it was too wide."

Now "it" means the street. Same position, same word, completely different meaning — resolved only by looking at other words ("tired" vs "wide"). Any model that processes words in isolation, or that forgets earlier context, will fail here. The model needs a way to ask: "which other words should influence my understanding of this word?"

See it happen

Click any word below to see where it "looks". Line thickness and opacity show attention weight. Try clicking it in the first sentence — then switch sentences and explore. Toggle the heatmap view to see all words at once.

Sentence

“it” attends most to “animal”. Notice how "it" attends strongly to "animal" — the model resolves the pronoun. If the sentence ended in "too wide", it would attend to "street" instead.

That heat pattern is not decoration — it's a real (hand-crafted but realistic) picture of what trained attention heads do: pronouns attend to their referents, verbs attend to their subjects and objects, adjectives attend to what they modify.

How it works, minus the matrix algebra

Each word (token) produces three vectors, learned during training:

a Query — "here's what I'm looking for"
a Key — "here's what I contain"
a Value — "here's the information I'll hand over if you pick me"

To compute attention for one word, compare its Query against every word's Key (a dot product — the same similarity math from the embeddings module!). Big match → big score. Run the scores through a softmax so they become positive weights summing to 1. Then blend all the Values together using those weights. The result: a new representation of the word, enriched by exactly the context that matters.

scores  = query @ keys.T / sqrt(d)   # how relevant is each word to me?
weights = softmax(scores)            # normalize to probabilities (sum to 1)
output  = weights @ values           # weighted blend of everyone's info

Three lines. That's the mechanism behind the demo above and behind every frontier model.

Multi-head attention

Real models run many attention "heads" in parallel — 32, 96, or more per layer. Each head learns to track a different kind of relationship: one follows pronouns, another tracks syntax, another watches for quotes or code brackets. Their outputs are combined, then the whole thing is stacked dozens of layers deep. The heatmap you explored is what a single head in a single layer might look like.

Why this was such a big deal

Long-range connections for free. Word 1 and word 10,000 are one attention step apart. No more information decaying as it's passed along a chain.
Parallel training. Unlike older recurrent models that had to process tokens one-by-one, attention over a whole sequence computes at once — perfect for GPUs, which made trillion-token training runs feasible.
The cost. Every token attends to every other token: for n tokens that's n × n comparisons. Double the context, quadruple the work. This quadratic cost is the reason context windows are limited — the subject of the next lesson.

Build it yourself

Implement single-head attention in ~15 lines of NumPy: random Q, K, V matrices, the three-line computation above, and a matplotlib imshow of the softmax weights for a toy sentence. Then hand-tune the Q/K vectors so that "it" attends to "animal" — you'll appreciate exactly what training has to discover on its own.

Summary

Attention lets each token look at all other tokens and weight them by learned relevance.
Mechanism: compare Query vs Keys (similarity), softmax to weights, blend Values.
Multiple heads across many layers each track different relationships (pronouns, syntax, structure).
It enabled parallel training and long-range understanding — the Transformer breakthrough behind all modern LLMs.
Its quadratic cost in sequence length is why context windows have limits — next lesson.