distiliaa Learn
Module 5 15 min

Context Windows & Positional Encoding

How models know word order and why context is limited.

Attention has a strange blind spot: it treats the input as a bag of tokens. Compare every Query to every Key and you get the same result whether the sentence is "dog bites man" or "man bites dog". Yet word order is obviously crucial. This lesson covers the two practical realities that follow: how models learn position, and why every model has a hard limit on how much it can read — the context window.

Why does this exist?

Attention was designed to be order-blind on purpose — that's what makes it parallel and fast. But language depends on order, so engineers had to inject position information back in (positional encoding). And because attention compares every token with every other, longer inputs get quadratically more expensive — so every model ships with a maximum context length. These two constraints shape real systems: they're why prompts get truncated, why long chats "forget" their beginnings, and why RAG (Module 7) exists at all.

Order-blindness, concretely

Recall the attention math: scores = query @ keys.T. Nothing in a dot product knows whether a token was 3rd or 300th. Shuffle the input tokens and (without positional info) each token's attention output is identical — the model literally could not distinguish:

"The chef thanked the critic"     →  same attention results
"The critic thanked the chef"     →  as this. Yikes.

Fix: stamp every token with its position

The solution is to add a positional encoding to each token's embedding before attention runs — like numbering the pages of a shuffled manuscript.

  1. Token embeddingsEach token becomes a vector capturing its meaning (Module 3).
  2. Position signal
  3. Combine
  4. Attention runs

The classic Transformer used fixed sine and cosine waves of different frequencies — position 7 gets a unique wave-pattern fingerprint that also makes "3 tokens apart" look similar anywhere in the text. Modern models mostly use RoPE (rotary positional embeddings), which rotates the Query and Key vectors by an angle proportional to position, elegantly making attention depend on relative distance. You don't need the trigonometry — the takeaway is: position is injected data, not something attention natively knows.

The context window: the model's entire world

The context window is the maximum number of tokens the model can attend over — prompt and response combined. Some reference points (check current docs; these move fast):

| Era / model class | Context window | |---|---| | GPT-2 (2019) | 1,024 tokens | | GPT-3 (2020) | 2,048 tokens | | GPT-4 (2023) | 8K–32K tokens | | Claude 3 / Gemini 1.5 era (2024) | 200K–1M tokens |

Why not infinite? Attention's n × n cost: 10x the tokens ≈ 100x the attention compute, plus a growing KV cache (stored Keys/Values for every token) eating GPU memory. Long context is an active engineering battlefield — and positional encodings are part of it, since a model trained on 8K positions has never "seen" position 100,000.

Lost in the middle

A big context window is not a guarantee of attention quality. Research shows models recall information at the start and end of a long context much better than facts buried in the middle. Practical rule: put critical instructions at the beginning, the immediate question at the end, and don't assume page 400 of a dumped PDF was actually "read" carefully.

What this means when you build

  • Chats forget. A conversation that outruns the window gets truncated (usually oldest-first) or summarized. The model isn't "forgetting" — the text is literally no longer in its input.
  • Count your tokens. Remember Module 2: cost and limits are in tokens, not words. A 300-page book (~120K tokens) may simply not fit.
  • Don't stuff — retrieve. Cramming everything into context is slow, expensive, and dilutes attention. Retrieving only relevant chunks is usually better — that's RAG, coming in Module 7.
# Every serious LLM app has a version of this check
tokens = count_tokens(system_prompt) + count_tokens(history) + count_tokens(question)
if tokens > CONTEXT_LIMIT - MAX_RESPONSE_TOKENS:
    history = summarize_or_trim(history)   # something must give

Build it yourself

Write a "context budget" helper: given a system prompt, a list of chat messages, and a limit, it drops oldest messages (or replaces them with a one-line summary) until everything fits, always keeping the system prompt and latest user message. Log what got dropped. This tiny utility exists inside virtually every production chatbot.

Summary

  • Attention is order-blind; positional encodings (sinusoidal, learned, RoPE) inject word order back in.
  • The context window is the hard limit on tokens the model can consider — prompt plus response.
  • Limits exist because attention is quadratic and the KV cache eats memory.
  • Even huge windows suffer lost-in-the-middle: position your important content deliberately.
  • Managing the context budget (trim, summarize, retrieve) is core AI engineering — and the motivation for RAG.