The Intelliwise

The Intelliwise — lunch-box note

← Back to the lunch-box map

LoRA — longer guide

Freeze the big textbook. Train two thin matrices whose product is a small update.

This is the famous member of the cheap fine-tuning box on the lunch-box map. PEFT is the family name. LoRA is the small add-on most teams actually ship.

What problem it solves

Keep the whole textbook; only rewrite a sticky note.

Full fine-tuning writes a new copy of every weight. For a 7B–70B model that is money, disk, and a merge nightmare if you want many specialists. Hu et al. (2021) freeze the pretrained matrix W0 and learn a low-rank update BA beside it. Same task quality on their GPT-2 / GPT-3 / RoBERTa / DeBERTa / CLIP-style tests, far fewer trainable numbers.

Full fine-tune LoRA
Every weight moves W0 frozen; only A and B train
One full checkpoint per skill One base + a small adapter file per skill
Serve is “just the model” You can merge W0 + BA and serve as one

How the idea works

Pick a big linear map inside the net — the paper’s favourite targets are attention projections (query / value, sometimes more).

  1. Leave W0 as it was at pretraining.
  2. Add a low-rank update ΔW = BA, where B is tall-and-thin, A is short-and-wide, and the inner size is rank r (often 4, 8, 16…).
  3. Scale the update (the paper’s α / r habit). That is the volume knob.
  4. Run the forward pass as a frozen path plus the small update (formula below).
  5. At deploy time, fold BA into W0 if you want zero extra latency. Or keep adapters unmerged and hot-swap files.
\[h = W_0 x + BA x\] 
flowchart LR
  X[input] --> W[frozen W0]
  X --> A[thin A]
  A --> B[thin B]
  W --> Y[sum]
  B --> Y

Tiny sticky notes instead of rewriting the whole book

Why “low rank” is a bet. The paper’s claim: the change you need for a downstream task lives in a small subspace. You do not need a full d × d delta. If the bet is wrong for your task, raise r — or admit you wanted a full fine-tune.

Simple picture: the printed textbook stays on the shelf. A new course is a few notes on the hard pages. Swap the notes; the book stays.

Why it mattered / what it unlocked

  • Default “we fine-tuned it.” In a lot of real stacks, that sentence means LoRA (or QLoRA: same small update, 4-bit base).
  • Many skills, one backbone. Support tone, internal jargon, a tool habit — three files, not three 13B copies.
  • Fits the PEFT plugin story. Hugging Face PEFT is where most engineers import this. The paper is the maths; the library is the door.
  • Mergeability. Unlike some older adapter stacks, you are not forced to pay extra depth at serve if you bake the update in.

This site already has a longer engineer tour of the whole family: PEFT, LoRA, and QLoRA. This page stays on Hu et al.

What to remember

  • Frozen W0 + BA with small rank r.
  • Target modules matter (often q_proj / v_proj; more modules → more capacity).
  • α and r are the two knobs people actually turn.
  • Merge to serve cheap; keep unmerged to swap skills.
  • LoRA is one PEFT method. If someone says “we did PEFT,” ask which.

When to read the real paper

Open Hu et al. when you want:

  • the rank / α ablations and which layers they adapt
  • GPT-2 / GPT-3 / NLU tables vs full fine-tune and vs other adapters
  • the “intrinsic dimension” intuition they lean on
  • why they argue merge has no extra inference latency

Skip the grind if you only needed “small update, freeze the book.”

No special “LoRA dataset.” They reuse ordinary downstream sets from the models they adapt.

Back on the tray: lunch-box map · PEFT