distiliaa Learn
Module 11 12 min

When to Fine-Tune

Prompting vs RAG vs fine-tuning — picking the right tool.

By now you've seen three ways to make a model do what you want: write better prompts, give it retrieved context (RAG), and — the topic of this module — change the model's weights themselves. Fine-tuning is the most powerful and the most expensive of the three, and the single most valuable skill here is knowing when not to use it.

Why does this exist?

Pre-trained models are generalists. Sometimes generalist behavior isn't enough: you need a specific tone, a strict output format, deep fluency in a niche domain, or a small cheap model that mimics an expensive one. Fine-tuning exists to bake behavior into the weights — so you stop paying for it in every prompt.

The three levers

Think of them as escalating commitments:

  1. Prompting — change the instructions. Costs minutes. Ship today.
  2. RAG — change the knowledge available at runtime. Costs days. Knowledge stays fresh because you update the index, not the model.
  3. Fine-tuning — change the model itself. Costs weeks: you need training data, GPUs (or a tuning API), evaluation, and ongoing maintenance of a model artifact.

A useful mental model:

  • Prompting teaches the model what you want right now.
  • RAG teaches it what is true right now.
  • Fine-tuning teaches it how to behave, permanently.

The decision framework

Ask these in order — most projects stop at the first or second question.

1. Have you actually maxed out prompting? Few-shot examples, a precise system prompt, and structured output constraints solve a shocking fraction of "we need fine-tuning" requests. If you haven't tried ten serious prompt iterations with an eval set, you're not done.

2. Is the problem missing knowledge? Facts about your product, fresh documents, user-specific data — that's RAG territory. Fine-tuning is a terrible way to store facts: models trained on your docs still hallucinate, and every doc update means retraining. Retrieval gives you citations and instant updates.

3. Is the problem behavior, form, or skill? This is where fine-tuning earns its keep:

  • Style and tone — always answer like your brand, in your format, in your language register.
  • Strict output structure — always emit valid, domain-specific JSON/XML/DSL without burning prompt tokens on examples.
  • Domain fluency — medical notes, legal citations, your internal codebase's idioms — where the patterns matter more than lookup-able facts.
  • Distillation — teach a small, cheap, fast model to imitate a large model's outputs on your specific task. Often the strongest economic case: same quality on your narrow task at a tenth the latency and cost.
  • Prompt compression — if you're stuffing 3,000 tokens of instructions and examples into every request, fine-tuning can bake them in and slash per-request cost.

4. Do you have the data? Fine-tuning needs hundreds to tens of thousands of high-quality input→output examples. "High-quality" is doing heavy lifting: a model trained on mediocre examples reliably reproduces mediocrity. No data, no fine-tune — go collect or generate some first.

Fine-tuning does not fix hallucination

A model fine-tuned on your documentation will still invent plausible-sounding nonsense — it just hallucinates in your brand voice. Grounding answers in retrieved sources (RAG) is the hallucination mitigation; fine-tuning is not.

They stack

This isn't a competition. Production systems routinely combine all three: a fine-tuned small model (behavior) + RAG (knowledge) + a tight system prompt (task framing). A support bot might use a fine-tuned model for tone and ticket-format compliance, retrieve help-center articles for facts, and prompt-inject the customer's plan details.

The costs nobody mentions

  • Evaluation burden. You now need eval suites to prove the tuned model beats the base model — and to catch regressions on general ability (tuning can make models worse at everything outside the training distribution; this is called catastrophic forgetting).
  • Operational drag. You own a model artifact: versioning, redeployment when the base model updates, re-tuning when your task drifts.
  • Iteration speed. A prompt change ships in minutes; a fine-tune iteration takes hours to days.

The one-line rule

Reach for fine-tuning when you can say: "I have thousands of great examples of exactly the behavior I want, prompting plateaued below the quality bar, and the behavior — not the knowledge — is the problem."

Build it yourself

  1. Pick a task you've prompted before (e.g. summarizing tickets into a fixed JSON schema).
  2. Write 20 gold input→output examples by hand — feel how expensive good data is.
  3. Evaluate your best prompt against those 20 examples. Score format compliance and quality.
  4. Only if the prompt genuinely plateaus: you now have the seed of a fine-tuning dataset — and you'll learn how the training works cheaply in the next lesson.

Summary

  • Prompting changes instructions, RAG changes available knowledge, fine-tuning changes behavior in the weights.
  • Fine-tune for behavior, format, style, domain fluency, and distillation — not for storing facts.
  • It does not cure hallucination and it adds real costs: data curation, evals, and owning a model artifact.
  • The techniques stack; most strong production systems use all three.
  • Prerequisites: prompting has measurably plateaued, and you have (or can build) a high-quality example dataset.