LoRA & QLoRA

So you've decided fine-tuning is the right tool. Bad news first: fully fine-tuning a 7-billion-parameter model means storing gradients and optimizer state for all 7 billion weights — north of 100 GB of GPU memory. Good news: a 2021 technique called LoRA lets you get most of the benefit while training well under 1% of the parameters, often on a single consumer GPU.

The core idea: don't change W, add to it

Take one weight matrix W in the model — say 4096×4096, about 16.8 million parameters. Full fine-tuning nudges all 16.8M values.

LoRA freezes W completely and learns a low-rank update instead:

W' = W + B·A

W : d×d      frozen  (16.8M params, untouched)
A : d×r      trainable
B : r×d      trainable

The rank r is tiny — typically 4 to 64. With d = 4096 and r = 8, the trainable parameters are 2 × 4096 × 8 = 65,536 — 0.4% of the original matrix. Multiplying B·A produces a full d×d matrix, but one whose "information content" (rank) is limited to r. The bet — which holds remarkably well in practice — is that task adaptation doesn't need a full-rank change.

At inference you can either keep the adapter separate (and swap adapters per task on one shared base model!) or merge B·A into W once for zero extra latency.

See it

The visualizer below shows one 12×12 weight matrix. In full fine-tune mode every cell pulses — all 144 parameters trainable. Switch to LoRA: W dims (frozen), and two thin strips A and B appear. Drag the rank slider and watch the trainable count and savings change.

Notice that even at rank 4, you train 96 parameters instead of 144 — and the savings grow with matrix size: for real 4096-wide matrices the ratio is under 1%.

QLoRA: quantize the frozen part

LoRA shrinks the trainable memory, but the frozen base model still sits in GPU memory in 16-bit precision — 14 GB for a 7B model. QLoRA's insight: since W is frozen anyway, why store it precisely? Quantize it to 4 bits (using a format called NF4), keep the LoRA adapters in full precision, and dequantize on the fly during the forward pass.

Toggle QLoRA in the visualizer above — the frozen matrix snaps to a coarser color palette. That's the visual metaphor for 4-bit storage: fewer distinct values, ~4× less memory, nearly the same fine-tuned quality. QLoRA is what made "fine-tune a 7B–70B model on one GPU" a normal weekend activity.

The code

With Hugging Face peft, LoRA is a config object:

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    load_in_4bit=True,          # <- the "Q" in QLoRA
)

config = LoraConfig(
    r=16,                       # rank of the update
    lora_alpha=32,              # scaling factor (roughly: how loud the adapter is)
    target_modules=["q_proj", "v_proj"],   # which matrices get adapters
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 8,036,929,536 || trainable%: 0.0848

Training then proceeds like any fine-tune — same loss, same optimizer — but gradients flow only into the A and B matrices. The output artifact is an adapter file of a few dozen megabytes, not an 16 GB model copy.

Build it yourself

1. Take the 20 gold examples you wrote in the previous lesson and expand to ~200 (hand-write, or generate with a big model and review each one).
2. QLoRA-tune a small open model (1–8B) with peft + bitsandbytes on a single GPU (Colab works).
3. Log print_trainable_parameters() — verify you're under 1% trainable.
4. Evaluate base vs tuned on 20 held-out examples. Also spot-check a few general questions to catch catastrophic forgetting.

Summary

• Full fine-tuning needs gradients + optimizer state for every weight — ~8× model size in memory.
• LoRA freezes W and trains a low-rank update B·A, typically under 1% of parameters, producing a tiny adapter file.
• Rank r controls adapter capacity; 8–16 is a strong default, raise only when evals demand it.
• QLoRA stores the frozen base in 4-bit, enabling large-model fine-tuning on a single GPU.
• Adapters merge for zero-latency inference, or stay separate for hot-swappable multi-task serving.