Inside the Transformer

You've met the pieces: tokens, embeddings, next-token prediction, attention. Now let's assemble the whole machine and — more importantly — take it apart. By the end of this lesson you'll be able to look at any LLM's spec sheet ("70B parameters, 80 layers, 64 heads, 8k context") and know exactly what each number means and what changing it would do.

The stack, top to bottom

A transformer is a short pipeline repeated many times:

1. Tokenizer — text becomes token IDs (Module 2, no learning involved).
2. Token embeddings — each ID becomes a learned vector (Module 4's geometry).
3. Positional encoding — each vector gets stamped with its position, so word order exists.
4. Transformer layers, repeated N times — each layer does two things:
   • Multi-head attention: every token gathers information from every other token.
   • Feed-forward network (FFN): each token's vector is processed independently through a small neural net.
5. LM head — the final vector is scored against the entire vocabulary.
6. Softmax + sampling — scores become probabilities; one token is chosen; the loop repeats.

That's the entire architecture. A 7B model and a 700B model differ only in how many layers they stack (depth) and how wide each vector is (width).

Break it and see

The explorer below has three parts. First, the architecture itself — one block per component, top to bottom, with the repeated layer clearly boxed. Click any block to read its job, and use the ON/OFF switches to remove components. Second, the two scale dials every transformer has. Third, the end-to-end result: one fixed prompt run through whatever configuration you've built.

Work through these experiments in order:

1. Switch off attention. Every token is now processed alone — no token can see its neighbors. The output collapses into frequency mush. This is the single most important ablation: attention is what makes context exist.
2. Attention on, positional encoding off. Now tokens see each other but not the order — the model reads a bag of words. "The cat chased the mouse" and "the mouse chased the cat" become the same input. Watch the output scramble word order.
3. FFN off. Attention keeps mixing information around, but nothing ever computes on it. Roughly two-thirds of a model's parameters live in the FFNs — this is where knowledge is stored.
4. All on, minimum scale. Slide layers and width down. The architecture is correct but tiny — watch it latch onto a phrase and repeat it. Small models don't fail randomly; they fail by looping.
5. Scale up and watch the parameter counter. Parameters grow with layers × width² — width is quadratic. Double the width and memory roughly quadruples. This is why "make it bigger" gets expensive so fast, and why the memory readout maps directly to GPU cost in Module 12.

Reading a real spec sheet

Now decode a typical model card with what you know:

| Spec | Meaning | Lever it pulls | |---|---|---| | 70B parameters | Total learned weights (embeddings + attention + FFNs) | Knowledge capacity, memory, cost | | 80 layers | The attention+FFN block repeated 80 times | Depth of reasoning per token | | d_model 8192 | Every token is an 8192-number vector | Nuance each token can carry | | 64 heads | Attention runs 64 ways in parallel per layer | Different relationship types tracked at once | | 8k context | Max tokens attention can span | How much the model can "see" at once |

Build it yourself

In the explorer, find the smallest configuration (fewest layers, narrowest width) that still produces the coherent sentence rather than the looping one. That boundary — where capability appears as scale crosses a threshold — is a miniature version of the "emergent abilities" you'll hear about in LLM discussions. Then, for the ablations: before flipping each switch, write down what you predict the output will look like. Getting these predictions right is what "understanding the architecture" means.

Summary

• A transformer is a short pipeline — tokenize → embed → add position → (attention + FFN) × N → score vocabulary → sample — with all the size in the repeated middle.
• Attention creates context, positional encoding creates order, FFNs store knowledge — each fails in its own recognizable way.
• Depth (layers) and width (d_model) are the two scale dials; parameters grow with layers × width², which is why width is expensive.
• Small models fail by looping, not by random noise — capability appears as scale crosses thresholds.
• You can now read any model spec sheet and know which lever each number pulls.