The Full RAG Pipeline

You know why RAG exists and how documents get chunked. Time to assemble the whole machine: document in, grounded answer out. Every production RAG system — from "chat with your PDF" toys to enterprise knowledge assistants — is a variation of the eight-stage pipeline in this lesson.

The two halves

A RAG system runs in two phases, at two different times:

• Ingestion (offline, ahead of time): Document → Chunking → Embeddings → Vector DB. You run this once per document, whenever content changes.
• Query (online, per question): Retriever → Prompt → LLM → Answer. This runs on every user question, in milliseconds-to-seconds.

The split matters operationally: ingestion bugs (bad chunking, stale index) poison every future answer, while query bugs (bad prompt template, wrong top-k) can be fixed instantly without reprocessing anything.

Walk the pipeline

Click each stage below to see what happens inside it. Use Next and Previous to walk through slowly, or hit Run query to watch the question move through the whole system — ending with the exact assembled prompt and the final answer.

A few things the animation makes concrete:

• The question gets embedded too. Retrieval works because the query and the chunks live in the same vector space — closeness in that space approximates "about the same thing."
• The retriever returns scores, not certainty. Top-3 chunks with similarities of 0.93, 0.71, 0.64 — the model will see all three, relevant or not. Garbage retrieval means garbage answers, no matter how good the LLM is.
• The prompt is just string assembly. System instructions + retrieved chunks + question. There's no hidden magic between the vector DB and the model — you literally paste text.

The same pipeline in code

Here's an honest miniature of the whole thing (using a generic embedding function and any chat model):

# ---------- INGESTION (run once per document) ----------
chunks = chunk_text(load("refund-policy.md"), size=500, overlap=75)
index = [(embed(c), c) for c in chunks]        # tiny "vector DB"

# ---------- QUERY (run per question) ----------
def answer(question: str, k: int = 3) -> str:
    q_vec = embed(question)
    top = sorted(index, key=lambda p: cosine(q_vec, p[0]), reverse=True)[:k]
    context = "\n".join(f"- {text}" for _, text in top)

    prompt = f"""Answer using ONLY the context below.
If the answer isn't in the context, say "I don't know."

Context:
{context}

Question: {question}"""
    return llm(prompt)

A real system swaps the list for an actual vector database (a later module), adds metadata filters and citations, and streams the answer — but the skeleton is exactly this.

Where pipelines break

When answers go wrong, debug stage by stage, upstream first:

• Bad chunking → the answer exists but is split across chunks, so no single retrieved passage contains it. Fix: sentence-aware splitting, overlap, or bigger chunks.
• Retrieval miss → the right chunk exists but doesn't rank in the top-k. Fix: better embedding model, more k, or hybrid search (next lesson).
• Prompt dilution → the right chunk was retrieved, alongside so much junk that the model latched onto the wrong passage. Fix: fewer/better chunks, reranking.
• Generation failure → perfect context, wrong answer. Now — and only now — is it a prompting or model problem.

Build it yourself

1. Take your chunker from the previous lesson and a small document (10-20 paragraphs).
2. Use any embedding API (or a local model like sentence-transformers) to embed the chunks into a plain Python list — no vector DB needed at this scale.
3. Implement answer() as above: embed the question, cosine-sort, take top-3, assemble the prompt, call an LLM.
4. Ask five questions you know the answers to. For every wrong answer, print the retrieved chunks and classify the failure: chunking, retrieval, dilution, or generation?

Congratulations — that classification exercise is the actual daily job of a RAG engineer.

Summary

• RAG = an offline ingestion pipeline (document → chunks → embeddings → vector DB) plus an online query pipeline (retrieve → prompt → LLM → answer).
• The query is embedded into the same space as the chunks; top-k nearest chunks get pasted into a prompt template.
• The final prompt is plain assembled text: instructions + context + question. No magic.
• Debug upstream-first: chunking → retrieval → dilution → generation, and always log retrieved chunks.
• The skeleton fits in 20 lines of code; everything else in production RAG is hardening this skeleton.