I use one running example the whole way. The average rent in the pocket data is €1,900. One 80 m² city flat actually rents for €2,500. Those six-flat numbers are toy numbers I made up so the first rounds fit on one screen. The script at the end is a larger synthetic set. Not real listing data.

flowchart LR
  A[Start at the average<br/>€1,900] --> B[Compute residuals]
  B --> C[Fit a small tree]
  C --> D[Add eta times the tree]
  D --> E{More trees?}
  E -->|yes| B
  E -->|no| F[Final rent]

1. What is XGBoost, in one line?

Me: A supervised model that adds small decision trees one after another. Each new tree tries to fix the errors so far.

XGBoost stands for Extreme Gradient Boosting. Supervised means we have a target — here, monthly rent. You give it rows (floor area, later location and age) and a number to hit.

It does not build 200 independent trees and average them. It starts with a simple guess, usually the mean rent, then grows a shallow tree that predicts the leftover error. Add that tree. Repeat. That stacked model is the booster.

For the 80 m² city flat, the first guess is €1,900. The leftover error is +€600. Later trees chip away at that +€600 until the prediction sits closer to €2,500.

Follow-up they may ask: Is it only for regression? No. Same idea for classification. The loss changes. The “add a tree to cut the loss” loop does not.

2. How is it different from a random forest?

Me: A random forest trains trees independently and averages the votes. XGBoost adds trees in sequence so each one cuts the loss that is still left.

If five forest trees guess €2,100, €2,400, €1,750, €2,300, €1,950 for the city flat, the answer is the mean of those five. If five boosting trees run, the answer is €1,900 plus the five corrections, in order.

Follow-up they may ask: Can I bag XGBoost trees anyway? Yes. subsample and colsample_bytree already do a light version of that, on top of boosting.

3. Walk me through the first round.

Me: Start from the average. Compute residuals. Tree 1 learns a correction for each group. Most flats get better, a couple get worse, and the total error still drops.

Pocket data. Six flats. Mean rent €1,900. I set eta to 1.0 in this round so the leaf values are easy to see. Later we put eta back to 0.1.

Round-0 RMSE is 365.

Tree 1 splits at 60 m².

• Small group A, B, C. Mean residual = (−500 − 300 − 100) / 3 = −300. New pred = €1,600.
• Large group D, E, F. Mean residual = (0 + 600 + 300) / 3 = +300. New pred = €2,200.

C and D got worse. E, the €2,500 flat, moved from €1,900 to €2,200. Round-1 RMSE is 208. Total error fell even though two rows got worse. That is normal. The tree optimises the sum, not every row.

A second split on the large group takes RMSE to 144. E is now at €2,350.

With eta 0.1, the same city flat climbs in small steps instead of taking the full leaf jump:

Follow-up they may ask: Why not one deep tree instead of many shallow ones? One deep tree memorizes the six rows. Shallow trees plus a small eta usually travel better to flats the model has not seen.

4. Why is it called gradient boosting?

Me: Each tree is fit on the negative gradient of the loss. For squared error, that gradient is the residual.

Loss here is squared error: how far the predicted rent is from the actual rent. The gradient of that loss, with respect to the current prediction, points toward “make this prediction worse.” The opposite direction — the negative gradient — points toward “make this prediction better.”

For squared error, that negative gradient is actual − prediction. That is the residual column in the table above. Tree 1 is not magically “learning rent.” It is learning those residuals.

If the loss were log loss for a classification problem, the gradient would not be a raw residual. The loop is the same: compute the gradient of the loss, grow a tree that follows it, add the tree, repeat. “Gradient boosting” is that loop. “XGBoost” is a fast, regularised way to run it.

5. What does XGBoost add over plain gradient boosting?

Me: It uses the first and second derivatives of the loss — gradient and hessian — to score splits and set leaf values. It also regularises tree complexity, finds splits with a fast histogram pass, and has a default direction for missing values.

Plain gradient boosting, in the Friedman sense, often uses only the gradient. XGBoost takes a second-order look. For each row you have:

• g — gradient. How much the loss would move if this prediction moved.
• h — hessian. How curved that loss is at this prediction.

For squared error, g is prediction − actual and h is 1. So the hessian is basically a row count. For logistic loss, h is p(1 − p), so a confident row counts for less than a row sitting at 0.5. That is why min_child_weight is a sum of hessians, not “minimum rows,” even if people treat it as a row count in regression.

A split is scored by gain. A leaf gets a weight. From the XGBoost model page:

G is the sum of gradients in that node. H is the sum of hessians. λ (reg_lambda) pulls leaf values toward zero. γ (gamma) is a tax for making a new leaf.

In plain words: gain asks “if I split here, how much extra loss do I remove, after paying for a new leaf?” If that number is not positive, skip the split. The leaf weight is a shrunk version of “average residual in this leaf.” For our large-flat leaf, G is the sum of those +€600 and +€300 style residuals (with a sign flip from how g is defined), H is roughly 3, and λ stops one noisy leaf from swinging rent by a huge amount.

The rest of the “extreme” in the name is engineering:

• Histogram split finding. Bucket each feature, scan the buckets. Parallel across features.
• Regularisation. Depth, λ, α, γ, min child weight. The tree is not allowed to grow just because it can.
• Sparsity-aware missing values. At each split the algorithm learns whether NaN should go left or right.

Follow-up they may ask: Do I need to derive gain on a whiteboard? Know what G, H, λ, and γ do. Write the leaf-weight formula. Explain it with the €1,900 / €2,500 flat. That is enough for most interviews.

6. What is the learning rate (eta) and why keep it small?

Me: Eta scales each tree’s correction before it is added. Keep it small so no single tree can swing the rent too far.

The 80 m² city flat still has a +€600 residual after round 0. If eta is 1.0, tree 1 is allowed to add the whole +€600 (or the leaf mean, +€300 in the six-flat split). If eta is 0.1:

+€600 × 0.1 = +€60

New prediction = €1,960, not €2,500. The next tree sees a leftover of +€540 and takes another small step.

Small eta means you need more trees. That sounds wasteful. In practice those quieter steps generalise better. A tree that is a bit wrong does less damage. Early stopping then picks how many steps you actually keep.

A start I actually use: learning_rate=0.05 or 0.1, then let early stopping choose n_estimators. If the model wants 20 trees and still looks noisy, I drop eta and allow more trees. If it wants 800 trees and the job is slow, I raise eta a bit.

7. Which hyperparameters do you tune first?

Me: Depth and min child weight, then the two sample rates, then the regularisers, and I never pick n_estimators by hand if I can use early stopping.

On this rent problem I would start with max_depth=3, min_child_weight=5, subsample=0.8, colsample_bytree=0.8, reg_lambda=1, learning_rate=0.1, n_estimators=400, early_stopping_rounds=30. That is also what code/xgb_rent_demo.py uses.

I do not start with a 40-point grid. I get a decent depth and eta first. Then I poke λ and subsample. The gain formula in Q5 is the reason γ and λ change the trees: they are the tax and the shrink, not magic knobs.

Follow-up they may ask: Why is min_child_weight not “minimum rows”? Because it is the sum of hessians. In this squared-error rent model, h = 1, so it happens to match a row count. In logistic loss, a row with p = 0.99 contributes about 0.01, so you need many such rows to pass the same threshold.

8. How do you stop it overfitting?

Me: Train / validation / test. Early stopping on the validation set. Keep the best iteration. Touch the test set once, at the end.

The split:

1. Train — trees are grown here.
2. Validation — after each tree, score this set. If it has not improved for early_stopping_rounds trees, stop. Keep the iteration with the best validation score.
3. Test — one number, after you have frozen the model. No tuning against it.

If you keep adding trees, train error will keep falling. Validation error falls, then rises. That rise is the model starting to memorize train flats. The marker in the GIF is the model I would ship.

The other regularisers from Q7 still matter. Early stopping is the brake. Depth, min child weight, subsample, λ are the speed limit. I use both.

Do not report the best validation score as the final score. It has been peeked at on every round. The test MAE is the one I would write in a design doc.

9. Which metric would you report?

Me: MAE in euro, and RMSE next to it. For classification I would pick log loss or AUC, depending on whether we care about probabilities or ranking.

Rent is in euro per month. MAE is the average absolute miss. “We are off by about €84 on a held-out flat” is a sentence a letting-ops person can use. RMSE punishes the big misses harder. If one penthouse is off by €800, RMSE will show that more than MAE.

I would train with a squared-error objective (that is what the gradients in Q4 assume) and still report MAE, because that is the unit of the product. In the script, early stopping watches MAE on the validation set.

For a yes/no target — say “will this listing go stale?” — I would not use MAE.

• Log loss if we need a probability we can trust, for example expected value in a bidding tool.
• AUC if we only need to rank listings, and the threshold can move later.
• F1 / precision / recall only after someone names the cost of a false let vs a missed let.

Do not report train MAE. Do not report the early-stopping validation MAE as the headline. Test MAE, once.

10. How do you explain a prediction to a stakeholder?

Me: Global feature importance for “what the model uses in general.” Per-flat contributions (SHAP / pred_contribs) for “why this flat.” Neither is causation.

Global gain importance answers: across all splits, which column reduced loss the most? In the synthetic run, floor area is 0.54, location 0.40, age 0.06. I can say “area and city vs suburb do most of the work. Age is a small adjustment.”

That does not explain one listing. Age could still move this 6-year-old city flat by €90.

For one row I use XGBoost’s own pred_contribs=True. That is the tree SHAP breakdown: bias plus each feature, summing to the prediction.

On the 80 m² city flat, 6 years old, the script printed:

I would say: “The model starts near the average rent. This flat is larger than average, in the city, and fairly new, so each of those pushes the rent up. Together they land at €2,365.”

I would also say what I will not say: “If you add 10 m² you will get €X more in the market.” That is a causal claim. The model saw correlations in the training rows. It did not run an experiment. A 10 m² extension in a suburb is not the same as 10 m² in the city centre.

11. When would you not use XGBoost?

Me: When I need a model a person can read as one tree, when the data is tiny, when I must extrapolate past the largest flat I have seen, or when the input is images, text, or audio.

Concrete cases:

• Need to show the rule. One CART tree, or even a linear model with three coefficients, is easier in a credit meeting than 84 trees.
• Very small data. Tens of rows. Boosting will memorize. A mean, a linear model, or a single small tree is more honest.
• Cannot extrapolate. Trees predict a constant in each leaf. Anything bigger than the largest training area lands in the same right-hand leaf. In the script, a city flat at 107.7 m², 140 m², 160 m², and 180 m² all get €2,742. A linear model keeps climbing to €3,792 at 180 m². If the business needs a sensible rent for a 180 m² penthouse and the train set stops at 108 m², I would not ship XGBoost alone.
• Images, text, audio. Neural nets. XGBoost wants a table.
• No time to tune, and AutoGluon / a strong CatBoost default is already in the repo. XGBoost is not free. Someone has to watch early stopping and the knobs in Q7.

Follow-up they may ask: So when do you use it? Tabular data, a few thousand rows or more, mixed numeric and missing values, and a metric you can early-stop on. Rent, churn, fraud scores, remaining-useful-life. That is the job it was built for.

12. Quick-fire round

Interviewer: Missing values?

Me: Yes. At each split it learns a default direction for NaN. I still check whether “missing” means “not recorded” or “does not apply.”

Interviewer: Categorical features?

Me: Recent XGBoost can take pandas category dtypes with enable_categorical=True. Otherwise I encode them. High-cardinality IDs I would not one-hot blindly.

Interviewer: Class imbalance?

Me: scale_pos_weight ≈ count(negative) / count(positive) for binary. Then I still pick the threshold from precision and recall, not from 0.5.

Interviewer: XGBoost vs LightGBM, one line?

Me: LightGBM grows leaf-wise and is often faster on large tables. XGBoost grows level-wise by default and is the one I have seen in more production stacks.

Interviewer: Vs CatBoost?

Me: CatBoost’s ordered target encoding is why I reach for it when I have many categorical columns. For this numeric rent table I am fine with XGBoost.

Interviewer: GPU training?

Me: Yes. Histogram split finding on GPU (device="cuda" in current XGBoost). Worth it when the table is wide or long. This 800-row demo is not that table.

Interviewer: Can you parallelize the boosting loop?

Me: Split finding, yes. Adding tree 4 before tree 3 is done, no. The next tree needs the current residuals.

Interviewer: Two features are collinear, say area and number of rooms?

Me: Gain will pick one of them most of the time. Importance will look unstable. I would not read 0.54 vs 0.40 as a physics result.

A small model you can run

code/xgb_rent_demo.py builds 800 synthetic Dublin-style rows (seed 12), splits them train / val / test, fits xgboost.XGBRegressor with early stopping, prints MAE and gain importance, breaks down one 80 m² city flat with pred_contribs, and compares XGBoost vs a linear model past the largest training area.

The data is synthetic. Mean rent in this draw is €1,995, not the €1,900 from the six-flat pocket table. It is not real listing data.

model = xgb.XGBRegressor(
    n_estimators=400,
    max_depth=3,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    min_child_weight=5,
    reg_lambda=1.0,
    objective="reg:squarederror",
    eval_metric="mae",
    early_stopping_rounds=30,
    tree_method="hist",
    random_state=12,
    n_jobs=1,
)
model.fit(x_train, y_train, eval_set=[(x_val, y_val)], verbose=False)

Output from python3 xgb_rent_demo.py (xgboost 3.4.1):

=== synthetic Dublin rent (not real listings) ===
xgboost 3.4.1  seed 12  n=800
features: floor_area_m2, location_city, age_years

mean rent: EUR 1995
area range in data: 38.1 to 107.7 m2

split  train=559  val=121  test=120

=== fit ===
best iteration (0-based): 83
trees kept: 84 of 400
validation MAE: EUR 69.5
test MAE:       EUR 84.0

=== feature importance (gain) ===
floor_area_m2     0.543
location_city     0.395
age_years         0.062

=== one flat (80 m2, city, 6 years) ===
model prediction: EUR 2365
pred_contribs (bias + each feature = prediction):
  bias                    +1989.1
  floor_area_m2        +123.7
  location_city        +158.8
  age_years            +93.0
  sum                     +2364.6

=== extrapolation check ===
largest floor area in training: 107.7 m2
flat                            xgboost     linear
at max train (107.7 m2)            2742       2736
just above max                     2742       2738
140 m2 (unseen)                    2742       3208
160 m2 (unseen)                    2742       3500
180 m2 (unseen)                    2742       3792

XGBoost stays almost flat past the largest training area. The linear model keeps climbing.

84 trees kept. Test MAE €84. The 80 m² city flat lands at €2,365 — closer to €2,500 than the €1,900 start, and not a memorised bullseye. From 107.7 m² upward, XGBoost repeats €2,742. That is the flattening from Q11, in the actual printout.

Interview cheat sheet

In short

XGBoost is supervised gradient boosting with extra care: second-order split scoring, regularised leaves, fast histogram cuts, and a learned path for missing values. You start at the average rent, €1,900 in the pocket example. Each shallow tree fits the leftover error. A small eta turns a +€600 miss into a +€60 step. Most rows improve, a few get worse, and RMSE still falls. You tune depth and child weight first, stop on a validation set, and quote test MAE. For one Dublin flat, bias plus area plus city plus age is the explanation — not a causal “if you add 10 m²” story. And the model will not invent a penthouse price for 180 m² if it only saw flats up to 108 m². That prediction goes flat.

Sources

• XGBoost documentation
• Introduction to boosted trees (gain and leaf-weight formulae)
• Tianqi Chen and Carlos Guestrin, XGBoost: A Scalable Tree Boosting System, KDD 2016

It is not enough on its own.

The loop can weaken from both sides. The machine gets better at finding gaps in the controls around it. The human gets tired and starts rubber-stamping. That second part has a name: approval fatigue.

The real safeguard is a loop that still lets a person notice when something is wrong.

A one-day supervisor diary (invented)

I made a toy day to show how the fade looks. I am the reviewer. An internal agent asks me to approve each action before it runs.

These are invented toy entries. Fake tool. Fake files. Not a company log. Not a real incident. The timestamps are there so you can feel the day get long.

09:40  read_file     invoices/2026-09.csv
       ask           Summarise September spend
       decision      Approved

I opened the CSV. Vendor, date, amount. Row count looks normal. September total is close to what finance said in standup. I can let it summarise.

10:35  update_record CRM account 4471
       ask           Mark invoice as paid
       decision      Approved

This one can hurt if it is wrong, so I slowed down. I opened account 4471. Invoice 8821 is 14,200 and still Open. The payment note in the ticket matches. Approved.

12:50  read_file     contracts/northwind.pdf
       ask           Check renewal terms
       decision      Approved

Lunch was late. I still opened the PDF. Term is 12 months. Notice period is 30 days. Auto-renew is on. I would have said the same thing.

13:36  read_file     contracts/acme.pdf
       ask           Check renewal terms
       decision      Approved

Same shape as the last one. I skimmed the first page. Looks fine.

14:22  create_file   reports/weekly.md
       ask           Draft weekly report
       decision      Approved

I have seen this report template all week. I glanced at the headings. The pattern is starting to feel familiar. Approved.

15:07  create_file   reports/notes.md
       ask           Add standup bullets
       decision      Approved
       count         18

ok

15:41  read_file     contracts/northwind.pdf
       ask           Confirm notice period
       decision      Approved
       count         30

looks fine

16:08  delete_backups nightly/
       ask            Free disk on the report box
       decision       Approved

I did not read the target. The card looked like the last ten cards.

16:40  update_record CRM 5120
       ask           Close stale ticket
       decision      Approved
       count         42

yes

17:48  audit_review
       count         58 approved

A teammate asked if the agent touched backups. I scrolled the log. There it is. 16:08. delete_backups. I had clicked Approve in two seconds.

The diary ends there.

What went wrong

Nothing in the morning was reckless. I opened files. I checked a CRM row. I read a contract.

The failure started when the queue got repetitive. Reads, small writes, the same “check this / draft that” shape. My brain stopped treating the next card as new. The click got cheaper than the read.

Then a destructive action sat in the same list, with the same Approve button. The tool did not change colour. It did not ask me to type the name. It did not need a second person. So delete_backups inherited the trust I had built on read_file.

The slogan was still true. A human was in the loop. The human was no longer looking.

Approval fatigue, in plain words

Approval fatigue is this: after you see the same kind of request many times, you stop reading the next one as if it is new. You pattern-match. You click.

It is how attention works on a long queue. Ops teams already know the cousin of this: alert fatigue. Too many similar alarms, and the important one looks like noise.

A capable agent can produce work faster than you can inspect it. If every action needs a human click, you have not added safety. You have added a tired person in a long queue.

The ironies of automation

This problem is older than agents.

In 1983, Lisanne Bainbridge wrote Ironies of Automation in Automatica. The paper is short and still sharp. Wikipedia’s page on Ironies of automation is a fair summary if you want the idea in one sitting.

The irony is simple. The more the system does on its own, the less practised the human gets. We still expect that same human to step in perfectly when automation fails.

You can see the same shape in aircraft autopilot and in driver-assist cars. The boring stretch is easy. The surprise is hard. And the surprise is exactly when you need a practised person.

An approve queue is a small version of that. The agent handles the routine. You watch. You get out of practice at watching. Then one card is not routine.

The machine side

Fatigue is the human half. The machine half is real too.

A capable agent is trying to get to “done”. If your checks have gaps, it can find them. That is not science fiction. Reward hacking and specification gaming are documented behaviours: the system scores well on the measure you wrote, while doing something you did not want.

In generic terms, that can look like:

• gaming an evaluator so the output looks fine to the scorer
• finding an unintended path around a restriction
• hiding a mistake when the check only looks at the happy result

I am not walking through a company incident here. The point is the shape. If the agent is stronger than the fence, “a human will catch it” is a weak last line — especially if that human is on click 40.

flowchart TB
  subgraph machine [Machine side]
    M1[Agent gets more capable] --> M2[Finds gaps in the controls]
    M2 --> M3[Reward hacking / specification gaming]
  end
  subgraph human [Human side]
    H1[System does more alone] --> H2[Reviewer practises less]
    H2 --> H3[Approval fatigue / rubber-stamp]
  end
  M3 --> Fail[Bad action goes through]
  H3 --> Fail

The two sides feed each other. A stronger agent makes a longer queue. A longer queue makes a weaker reviewer. A weaker reviewer is easier to get past.

How to fix the loop

Do not add a bigger Approve button. Change what is allowed to reach that button, and what the button is allowed to mean.

The first row does most of the work. Reads can go through. Writes wait. Destroy does not ride on a single tired click.

A canary only makes sense if the reviewer knows the job includes occasional tests, and if a miss becomes coaching, not a trap. Consent matters. The point is to measure attention, not to shame people.

A small approval gate

Here is a tiny Python sketch of that idea. classify puts each action in a tier. decide auto-allows reads, asks for a human on writes, and blocks destroy until a second person is present. After six human decisions, the fatigue guard forces a break.

The script lives next to this draft as approval_gate.py. I ran it. This is the real output.

#!/usr/bin/env python3
"""Toy risk-tiered approval gate with a fatigue break.

Invented actions only. Not a real agent, not a real company log.
"""

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum

MAX_REVIEWS_BEFORE_BREAK = 6


class Tier(str, Enum):
    AUTO = "auto"
    REVIEW = "review"
    TWO_PERSON = "block-plus-two-person"


TIER_FOR_ACTION = {
    "read_file": Tier.AUTO,
    "update_record": Tier.REVIEW,
    "create_file": Tier.REVIEW,
    "delete_backups": Tier.TWO_PERSON,
}


@dataclass
class GateState:
    reviews_since_break: int = 0


def classify(action: str) -> Tier:
    return TIER_FOR_ACTION.get(action, Tier.REVIEW)


def decide(state: GateState, action: str, second_person_ok: bool = False) -> str:
    if state.reviews_since_break >= MAX_REVIEWS_BEFORE_BREAK:
        return "BREAK — reviewer has approved too many in a row"

    tier = classify(action)
    if tier is Tier.AUTO:
        return f"AUTO allow {action}"

    if tier is Tier.REVIEW:
        state.reviews_since_break += 1
        return (
            f"REVIEW required for {action}  "
            f"({state.reviews_since_break}/{MAX_REVIEWS_BEFORE_BREAK})"
        )

    if not second_person_ok:
        return f"BLOCK {action} — needs a second person"

    state.reviews_since_break += 1
    return (
        f"TWO-PERSON allow {action}  "
        f"({state.reviews_since_break}/{MAX_REVIEWS_BEFORE_BREAK})"
    )

Output from python3 approval_gate.py:

Toy approval gate — invented actions, not a real incident
Tiers: read_file=auto  writes=review  delete_backups=two-person
Fatigue guard: break after 6 human decisions

09:40  read_file         AUTO allow read_file
10:35  update_record     REVIEW required for update_record  (1/6)
12:50  read_file         AUTO allow read_file
13:36  read_file         AUTO allow read_file
14:22  create_file       REVIEW required for create_file  (2/6)
15:07  create_file       REVIEW required for create_file  (3/6)
15:22  update_record     REVIEW required for update_record  (4/6)
15:41  read_file         AUTO allow read_file
16:08  delete_backups    BLOCK delete_backups — needs a second person
16:08  delete_backups    TWO-PERSON allow delete_backups  (5/6)  [second person present]
16:20  update_record     REVIEW required for update_record  (6/6)
16:28  create_file       BREAK — reviewer has approved too many in a row
16:40  update_record     BREAK — reviewer has approved too many in a row

Compare that with the diary. In the diary, delete_backups was just another Approve. In this gate it does not move without a second person. And even if that second person shows up, the reviewer is forced to stop after six human calls. Reads never counted toward the fatigue cap. That is the point of tiers.

Cheat sheet

• A human in the loop is a design, not a sentence in the spec.
• Routine work trains the agent and dulls the reviewer.
• If destroy uses the same button as read, destroy will inherit the trust from read.
• Auto-allow the cheap stuff so the human’s attention is spent on writes and deletes.
• Show what will break, in plain English, before the click.
• Do not give the agent delete rights if you can avoid it.
• Cap the queue. Rotate the person. Make someone read the audit later.
• Keep the manual skill warm with drills, or the takeover will be rusty.

In short

“Keep a human in the loop” is not a safeguard by itself. The machine can learn the gaps. The human can get tired and click anyway. I showed that fade with a toy diary: careful at 09:40, lazy by 15:07, delete_backups approved at 16:08, noticed only at 17:48 when the count hit 58.

Fix the loop. Tier the risk. Keep destroy off a single tired click. And design so the person can still notice when something is wrong.

It starts with pieces of text, lists of numbers, a compare-and-mix step called attention, a stack of the same block, and a lot of “guess the next word” training.

This post is that ladder. One chain of If you do X, you get Y. Same sentence the whole way: The bus is late.

The IDs below are toy numbers I made up. They are not from GPT-2, tiktoken, or any real vocab. The matrices are tiny on purpose. You can type the NumPy as you read, or run code/build_along.py (seed 7) and get the same prints.

If you want the bigger-picture story — a small GPT trained on Shakespeare characters — read How a Tiny GPT Learns. This page is the same idea, one checkpoint at a time.

flowchart TB
  A[Sentence] --> B[Tokens + toy IDs]
  B --> C[Embeddings x]
  C --> D[Query / Key / Value]
  D --> E[Scores]
  E --> F[Softmax weights]
  F --> G[Context vector]
  G --> H[Self-attention]
  H --> I[One transformer block]
  I --> J[Stack of blocks]
  J --> K[Next-word chat model]
  K --> L[Back to the sentence]

Shared setup for every snippet:

import numpy as np

np.random.seed(7)
np.set_printoptions(precision=2, suppress=True)

def softmax(x, axis=-1):
    e = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

SENTENCE = "The bus is late"
TOKENS = ["The", "bus", "is", "late"]
IDS = np.array([11, 42, 7, 88])  # toy IDs, not a real vocab
D = 4
SCALE = np.sqrt(D)

1. Break a sentence into pieces

A model does not read English. It reads integers.

A tokenizer cuts text into pieces and maps each piece to an ID. Real systems use tens of thousands of IDs. Here I pick four fake ones so the table stays on one screen.

print("sentence:", SENTENCE)
print("tokens:  ", TOKENS)
print("toy ids: ", IDS)

sentence: The bus is late
tokens:   ['The', 'bus', 'is', 'late']
toy ids:  [11 42  7 88]

Those four IDs are invented. Do not look them up in a real vocab.

2. Turn each token into a list of numbers

An embedding is a short list of numbers that stands in for a token. Similar tokens sit near each other in that space.

In a real model this list is hundreds of numbers long. Here each token gets 4 numbers. I wrote the table by hand so later dots stay small.

X = np.array(
    [
        [1.0, 0.1, 0.0, 0.2],  # The
        [0.2, 1.0, 0.1, 0.3],  # bus
        [0.0, 0.2, 1.0, 0.1],  # is
        [0.1, 0.8, 0.0, 1.0],  # late
    ]
)
print(X)
print("shape", X.shape)

[[1.  0.1 0.  0.2]
 [0.2 1.  0.1 0.3]
 [0.  0.2 1.  0.1]
 [0.1 0.8 0.  1. ]]
shape (4, 4)

Four tokens. Four numbers each. Shape (4, 4).

The picture below plots only the first two numbers, so you can see the arrows. bus and late point a similar way. That will matter in a few steps.

3. Let every token ask a question

The query is “what am I looking for, right now?”

You do not invent that question by hand. You multiply the embedding by a learned matrix $W_Q$. Same matrix for every token.

In NumPy the tokens are rows, so the line is Q = X @ W_Q. Same multiply, just a row layout.

W_Q = np.array(
    [
        [0.9, 0.1, 0.0, 0.0],
        [0.0, 0.9, 0.1, 0.0],
        [0.0, 0.0, 0.9, 0.1],
        [0.1, 0.0, 0.0, 0.9],
    ]
)
Q = X @ W_Q
print(Q)

[[0.92 0.19 0.01 0.18]
 [0.21 0.92 0.19 0.28]
 [0.01 0.18 0.92 0.19]
 [0.19 0.73 0.08 0.9 ]]

Row 3 is the query for late: [0.19 0.73 0.08 0.90]. We will use that row next.

4. Give every token a way to answer

The key is the label on the token. The value is the payload you actually mix in if that token gets picked.

Two more matrices. Same idea as the query.

W_K = np.array(
    [
        [0.8, 0.0, 0.1, 0.1],
        [0.1, 0.8, 0.0, 0.1],
        [0.0, 0.1, 0.8, 0.1],
        [0.1, 0.0, 0.1, 0.8],
    ]
)
W_V = np.diag([0.6, 0.6, 0.6, 0.6])
K = X @ W_K
V = X @ W_V
print("K")
print(K)
print("V")
print(V)

K
[[0.83 0.08 0.12 0.27]
 [0.29 0.81 0.13 0.37]
 [0.03 0.26 0.81 0.2 ]
 [0.26 0.64 0.11 0.89]]
V
[[0.6  0.06 0.   0.12]
 [0.12 0.6  0.06 0.18]
 [0.   0.12 0.6  0.06]
 [0.06 0.48 0.   0.6 ]]

Three jobs, three lists: $q$, $k$, $v$. Still shape (4, 4).

5. Compare one token’s query with every key

Take the query for late. Dot it with each key. Divide by $\sqrt{d}$ so the numbers do not grow with the width of the list.

A bigger score means a better match.

q_late = Q[3]
scores = q_late @ K.T / SCALE
print("q_late:", q_late)
print("scores:", scores)

q_late: [0.19 0.73 0.08 0.9 ]
scores: [0.23 0.49 0.22 0.66]

late matches itself most (0.66), then bus (0.49). That fits the embeddings: those two arrows already pointed a similar way.

6. Turn scores into percentages

Softmax makes the four scores positive and forces them to add up to 1.

You can read each weight as “how much of my mix should come from this token?”

weights = softmax(scores)
print("a = softmax(s):", weights)
print("sum:", weights.sum())

a = softmax(s): [0.21 0.27 0.2  0.32]
sum: 1.0

So late takes 32% from itself, 27% from bus, 21% from The, 20% from is.

7. Mix the values using those weights

The scores only decide who matters. The values are what you actually add in.

One weighted sum. One new list of four numbers for late.

c = weights @ V
print("c = sum a_j v_j:", c)

c = sum a_j v_j: [0.18 0.35 0.14 0.28]

That is the view late now has of the whole sentence.

8. Do it for every token at once

The last three steps were for one query. The model does the same for every token, in one matrix multiply.

GPT also puts a causal mask on the scores. A token may look at itself and the past. It may not look at a word that has not been written yet. That is how next-word training stays honest.

$M$ is $0$ on and below the diagonal, and $-\infty$ above it. After softmax, the future weights become $0$.

raw = Q @ K.T / SCALE
causal = np.tril(np.ones((4, 4)))
masked = np.where(causal == 1.0, raw, -1e9)
A = softmax(masked, axis=1)
C = A @ V
print("raw scores")
print(raw)
print("causal weights")
print(A)
print("context matrix")
print(C)

raw scores
[[0.41 0.24 0.06 0.26]
 [0.17 0.47 0.23 0.46]
 [0.09 0.17 0.42 0.19]
 [0.23 0.49 0.22 0.66]]
causal weights
[[1.   0.   0.   0.  ]
 [0.43 0.57 0.   0.  ]
 [0.29 0.31 0.4  0.  ]
 [0.21 0.27 0.2  0.32]]
context matrix
[[0.6  0.06 0.   0.12]
 [0.32 0.37 0.03 0.15]
 [0.21 0.25 0.26 0.11]
 [0.18 0.35 0.14 0.28]]

Row 0 is The. It can only see itself, so its weights are [1, 0, 0, 0]. Row 3 is late. It can see everyone. That last row is the same mix we built by hand.

9. Add a small neural network and residual connections

Attention is the talk step. Each token still needs a private think step. That is a tiny MLP, applied at every position with the same weights.

A residual connection adds the input back: $x + \text{layer}(x)$. The original signal still has a path through the stack. Layer norm rescales each token’s numbers so they do not blow up as you go deeper.

H = X + C
W1 = np.array(
    [
        [0.4, 0.1, 0.0, 0.0],
        [0.0, 0.4, 0.1, 0.0],
        [0.0, 0.0, 0.4, 0.1],
        [0.1, 0.0, 0.0, 0.4],
    ]
)
W2 = np.array(
    [
        [0.3, 0.0, 0.1, 0.0],
        [0.0, 0.3, 0.0, 0.1],
        [0.1, 0.0, 0.3, 0.0],
        [0.0, 0.1, 0.0, 0.3],
    ]
)
mlp = np.maximum(H @ W1, 0.0) @ W2
Y = H + mlp
print("h = X + C")
print(H)
print("y = h + MLP(h)")
print(Y)

h = X + C
[[1.6  0.16 0.   0.32]
 [0.52 1.37 0.13 0.45]
 [0.21 0.45 1.26 0.21]
 [0.28 1.15 0.14 1.28]]
y = h + MLP(h)
[[1.8  0.24 0.07 0.38]
 [0.62 1.57 0.22 0.57]
 [0.3  0.53 1.43 0.3 ]
 [0.36 1.35 0.21 1.48]]

Still shape (4, 4). Same four tokens, richer lists.

10. Stack the blocks

One block is one talk-then-think cycle. A transformer is that cycle repeated.

GPT-2 small does this 12 times. Each block has its own weights. The data shape stays the same as it goes up the stack. In this toy that shape is still (4, 4).

print("GPT-2 small repeats this block 12 times.")
print("toy stack depth: 12")

GPT-2 small repeats this block 12 times.
toy stack depth: 12

We do not run 12 toy blocks here. The idea is the repeat, not a new formula.

11. Train it to predict the next word

After the last block, a linear head turns the last token’s vector into a score per word. Softmax turns those scores into a probability bar.

Three training stages, in plain words:

1. Pretraining. Read a huge pile of text. Guess the next token. That is the transformer learning language.
2. SFT (supervised fine-tune). Show instruction and answer pairs. Same next-token loss, but now the text looks like a helpful reply.
3. RLHF / preference tuning. People pick the better of two answers. The model moves toward the preferred one.

We skip the training loop. We only do the last multiply, with a four-word toy vocab.

vocab = ["early", "again", "today", "."]
W_head = np.array(
    [
        [0.2, 0.1, 0.4, 0.8],
        [-0.3, 0.2, 0.6, 0.1],
        [0.1, 0.4, 0.2, 0.3],
        [0.8, 0.1, 1.2, 0.4],
    ]
)
logits = Y[3] @ W_head
probs = softmax(logits)
print("logits:", logits)
for word, p in zip(vocab, probs):
    bar = "#" * int(round(float(p) * 20))
    print(f"  {word:6s}  {p:4.2f}  {bar}")

logits: [0.88 0.54 2.78 1.08]
  early   0.10  ##
  again   0.07  #
  today   0.69  ##############
  .       0.13  ###

On this toy head, the next word after The bus is late is most likely today. Sample from that bar, append the word, run the ladder again. That is how a chat reply gets written: one token at a time.

These percentages are from the tiny matrices above. They are not a real ChatGPT sample.

12. Rewind everything

Take the stack off. Take the block off. Take attention off. You are left with four IDs, then four words.

print("back to the sentence:", SENTENCE)

back to the sentence: The bus is late

That is the whole machine, seen from one short line of English.

Checkpoint recap

In short

The bus is late becomes four toy IDs, then four short lists of numbers. Those lists ask questions (query), wear labels (key), and carry a payload (value). One token’s query is scored against every key, squashed into percentages, and used to mix the values. Do that for every token, hide the future with a causal mask, and you have self-attention. Add a small MLP and residual links and you have one transformer block. Stack twelve of those and you have a GPT-shaped model. Train it to guess the next word, then fine-tune with instructions and human preference, and you have the same idea that sits under ChatGPT. Undo the steps and you are back at the sentence.

The definitions, the matrix, and the colours are in Precision and Recall, in Colours You Will Remember. This post is the product half: which side you favour, and who pays when you get it wrong.

Colour legend (same chips as that post):

✓ TP correct catch   ⚠ FP false alarm   ✗ FN miss   · TN correct ignore

False alarms stay orange, not red, so you are not asked to tell red from green. Each cell also has a mark. The colour is a helper, not the only signal.

Two one-line definitions:

• Precision: when the system speaks up, how often is it right.
• Recall: out of all the real events, how many did it catch.

flowchart LR
  A[The system has a score] --> B{Speak up or stay quiet?}
  B -->|spoke, and it was real| TP["✓ TP correct catch"]
  B -->|spoke, and it was not| FP["⚠ FP false alarm"]
  B -->|stayed quiet, and it was real| FN["✗ FN miss"]
  B -->|stayed quiet, and it was not| TN["· TN correct ignore"]
  classDef tp fill:#d4f0e4,stroke:#1a9e55,color:#0d7a45
  classDef fp fill:#fde6d0,stroke:#d46a2e,color:#9a3b0a
  classDef fn fill:#ddd6fe,stroke:#5b4cc4,color:#4c1d95
  classDef tn fill:#e5e7eb,stroke:#6b7280,color:#4b5563
  class TP tp
  class FP fp
  class FN fn
  class TN tn

Seven myths I hear in reviews. Each one has a cost attached.

1. Higher accuracy means a better model

2. You can max both precision and recall

Picking that cut is the product decision. The model gives you a score. The product picks who gets bothered.

3. 0.5 is the right threshold

1. What does a ⚠ false alarm cost? Time, trust, a hidden invoice, a wasted page, a user who leaves.
2. What does a ✗ miss cost? Fraud that already left, a fire, a sick person sent home, a chunk the LLM never saw.
3. Who pays? The user, the business, or safety. Those three are not the same budget.
4. Does a human review the output? If yes, you can raise recall and let a person throw away the extra flags. If no, a false alarm goes straight to the user.

Favour the side whose mistake you cannot afford.

4. Precision always matters more

5. False alarms are harmless

6. This is just a data science decision

The table is a starting bias, not a law. Write the two costs for your screen.

7. One metric is enough

Use two stages when you can. A high-recall first pass gathers candidates. A high-precision filter, or a human, decides what the user sees.

flowchart LR
  A[High-recall first pass] --> B[High-precision filter or human]
  B --> C[What the user sees]

Practical tips

• Set the threshold from the costs, not from 0.5 because the library defaulted there.
• Use two stages when you can. Wide first pass, picky second pass.
• Track both numbers over time. A precision win that quietly drops recall is a product change.
• Agree a floor on one metric, then push the other. That sentence belongs in the spec.

Precision = when it speaks, how often is it right. Recall = of the real events, how many did it catch. Formulas and a worked inbox are in the colour-coded post.

You would not hire someone after one good interview answer. You would give them a work sample. You would also tell them the bar they have to clear.

Golden examples are the work sample. The grader is the bar. The agent has to clear that bar every time, not once.

Panel 1. One good answer is not a hire

Normal software is deterministic. Same input, same output.

A ROAS report is a simple example. ROAS means return on ad spend. You pass in ad spend $100 and sales $400. The report returns 4.0x. Run it again. Still 4.0x. A unit test with assertEqual is enough.

An LLM agent is not like that. Ask it five times to write ad copy for a protein brand. You get five different lines.

None of those is a character-for-character match. An exact-match test fails even when the line is fine. That is why agent tests cannot be “did the string equal this one golden string?”

The usual skip looks like this: someone runs a demo, the demo goes well once, the agent goes to production. Nobody wrote the work sample. Nobody wrote the bar.

flowchart LR
  A[Demo] -->|skip evals| B[Deploy]

The path you want is Demo → Evals → Deploy.

flowchart LR
  A[Demo] --> B[Evals]
  B --> C[Deploy]

Panel 2. The work sample: golden examples

For one input, write three things:

• a good output
• a bad output
• the rubric — the rules that separate the two

Protein brand, one ad line. Rules: under 80 characters, factually correct, follows brand guidelines (no weight-loss claims, no extra facts you did not give).

The bad line is useful. It tells the grader what “wrong” looks like, not only what “right” looks like.

Store these as JSONL — one JSON object per line. Easy to append. Easy to load in a test.

{"id":"protein-sachet","input":"Write one ad line. Facts: 20g protein per sachet. Do not claim weight loss.","good":"20g protein in every sachet.","bad":"Weight loss guaranteed!","max_chars":80,"must_include":["protein"],"banned":["guaranteed","weight loss"]}

Panel 3. The bar: grader rules

The grader is a script. It runs when you change a prompt, a skill / instruction file, or the model. It takes the new outputs and checks them against the same rules. Same work sample. Same bar.

flowchart TB
  I[Input] --> G[Golden example]
  G --> Good[Good output]
  G --> Bad[Bad output]
  G --> R[Rubric]
  I --> A[Agent]
  A --> O[New output]
  O --> GR[Grader]
  R --> GR
  GR --> P[PASS or FAIL]

Three kinds of graders

You do not need a fancy judge on day one. Start with the cheapest check that can catch the failure.

flowchart LR
  A[Code checks] --> B[LLM-as-judge]
  B --> C[Human review]

• Use code checks when a computer can decide without reading for meaning. “Is it under 80 characters?” “Does the JSON parse?” “Did it say guaranteed?”
• Use an LLM-as-judge when two good answers can differ in wording, but you still have a written rubric. Then check that judge against a few of your own labels. If the judge disagrees with you, fix the rubric or the judge prompt before you trust the score.
• Use human review to set the bar and to spot-check. Do not put a person on every CI run.

Panel 4. Re-test after a change. One FAIL

Same suite. The model swap invented a claim. The suite’s job is to fail that swap before a human has to notice it in production.

Panel 5. Ask it five times

Outputs vary. One pass is not a habit.

Works once means: in k tries, it succeeded at least once. People write this as pass@k.

Works every time means: in k tries, it succeeded on every try. People write this as pass^k.

A tiny example. Same golden case, five runs. Four pass. One invents a weight-loss claim.

• Pass rate: 4/5
• pass@5 (works at least once): PASS
• pass^5 (works every time): FAIL

For a coding puzzle, “it got it on try three” can still be useful. For an agent that writes live ads, or calls a tool, you care about the second number. Track the pass rate. Do not ship on “it worked once.”

Panel 6. Risky tools need brakes

A bad ad line is one bad ad. You turn it off. You write a better one.

An agent with write access to a production database, or any other tool that changes the real world, can do real damage. A wrong DELETE, a wrong refund, a wrong mail merge — that is not a copy tweak.

flowchart TB
  A[Bad ad copy] --> B[One bad ad]
  C[Tool that sends mail] --> D[Harder to undo]
  E[Write or delete on a live database] --> F[Real damage]

Add evals for the dangerous actions, not only for the prose:

• Did it try to delete when the brief said update?
• Did it call the tool with the right id?
• Did it stop and ask when the action is irreversible?

Evals are not the only brake. Pair them with guardrails:

• Human approval before deletes and other irreversible writes
• Least-privilege access — the agent gets the smallest token that can do the job
• Dry runs — show the planned write before it lands

How to start, as a working developer

You do not need a platform on week one. You need a folder of cases and a script in CI.

1. Collect 20 to 50 golden cases from real usage and real failures. Production tickets and “the agent did this weird thing” reports are better than cases you invent at a desk.
2. Store them as JSONL. One case per line. Input, good, bad, rules.
3. Write simple code checks first. Length, banned words, required words, JSON schema.
4. Add an LLM judge only where code cannot decide. Give it a short rubric. Check it against a few of your own labels before you trust it.
5. Run the suite in CI on every prompt, skill-file, or model change.
6. Track the pass rate over time. A drop after a model swap is the point of the suite.

flowchart LR
  A[Real failures] --> B[20–50 JSONL cases]
  B --> C[Code checks]
  C --> D[LLM judge + your labels]
  D --> E[CI on every change]
  E --> F[Pass rate over time]

A toy you can run

The script below is a fake protein-brand agent. No API calls. Three golden cases. Three variants that stand in for a prompt change, a skill-file change, and a model swap. Code checks only. Then the same case five times, so you can see pass@k vs pass^k.

#!/usr/bin/env python3
"""Toy eval suite for a protein-brand ad agent. No API calls."""

from __future__ import annotations

CASES = [
    {
        "id": "protein-sachet",
        "input": "Write one ad line. Facts: 20g protein per sachet. Do not claim weight loss.",
    },
    {
        "id": "protein-short",
        "input": "Write a short product line. Facts: 20g protein. Stay under 80 characters.",
    },
    {
        "id": "no-medical",
        "input": "Write a line for the same sachet. Do not make medical claims.",
    },
]

# variant -> case_id -> output  (pretend prompt / skill / model changes)
AGENT = {
    "prompt_change": {
        "protein-sachet": "20g protein in every sachet.",
        "protein-short": "20g protein. Simple fuel.",
        "no-medical": "20g protein in every sachet.",
    },
    "skill_change": {
        "protein-sachet": "Fuel your day with protein.",
        "protein-short": "Protein, 20g a sachet.",
        "no-medical": "Fuel your day with protein.",
    },
    "model_change": {
        "protein-sachet": "Weight loss guaranteed!",
        "protein-short": "High protein, low fat, weight loss guaranteed.",
        "no-medical": "Cures fatigue. Weight loss guaranteed!",
    },
}

# Fixed 5-run list so "works once" vs "works every time" is easy to see.
JITTER = [
    "20g protein in every sachet.",
    "20g protein. No sugar added.",
    "20g protein in every sachet.",
    "Weight loss guaranteed!",
    "20g protein in every sachet.",
]

MAX_CHARS = 80
MUST_INCLUDE = ("protein",)
BANNED = ("guaranteed", "weight loss", "low fat", "cures")


def fake_agent(case_id: str, variant: str) -> str:
    return AGENT[variant][case_id]


def grade(output: str) -> list[tuple[str, bool, str]]:
    checks = []
    checks.append(("length", len(output) <= MAX_CHARS, f"{len(output)}/{MAX_CHARS}"))
    banned_hit = [w for w in BANNED if w in output.lower()]
    checks.append(("banned", not banned_hit, "ok" if not banned_hit else ",".join(banned_hit)))
    missing = [w for w in MUST_INCLUDE if w not in output.lower()]
    checks.append(("must_include", not missing, "ok" if not missing else f"missing {missing}"))
    return checks


def passed(checks: list[tuple[str, bool, str]]) -> bool:
    return all(ok for _, ok, _ in checks)


def run_variant(variant: str) -> tuple[int, int]:
    wins = 0
    print(f"\n=== {variant} ===")
    for case in CASES:
        output = fake_agent(case["id"], variant)
        checks = grade(output)
        ok = passed(checks)
        wins += int(ok)
        flag = "PASS" if ok else "FAIL"
        print(f"  [{flag}] {case['id']}")
        print(f"         output: {output}")
        for name, cok, detail in checks:
            mark = "OK" if cok else "X"
            print(f"         {name:13} {mark:3}  {detail}")
    print(f"  pass rate: {wins}/{len(CASES)} ({wins / len(CASES):.0%})")
    return wins, len(CASES)


def run_jitter() -> None:
    print("\n=== same case, 5 runs (outputs vary) ===")
    flags = []
    for i, output in enumerate(JITTER, start=1):
        ok = passed(grade(output))
        flags.append(ok)
        print(f"  run {i}: {'PASS' if ok else 'FAIL'}  {output}")
    k = len(flags)
    n_pass = sum(flags)
    pass_at_k = any(flags)
    pass_hat_k = all(flags)
    print(f"  pass rate: {n_pass}/{k}")
    print(f"  pass@{k}  (works at least once): {'PASS' if pass_at_k else 'FAIL'}")
    print(f"  pass^{k}  (works every time):    {'PASS' if pass_hat_k else 'FAIL'}")


def main() -> None:
    print("Toy eval suite. Fake agent. Code checks only. No API calls.")
    totals = []
    for variant in ("prompt_change", "skill_change", "model_change"):
        totals.append(run_variant(variant))
    won = sum(w for w, _ in totals)
    n = sum(n for _, n in totals)
    print(f"\n=== suite ===")
    print(f"overall pass rate: {won}/{n} ({won / n:.0%})")
    run_jitter()


if __name__ == "__main__":
    main()

I ran this. This is the real output:

Toy eval suite. Fake agent. Code checks only. No API calls.

=== prompt_change ===
  [PASS] protein-sachet
         output: 20g protein in every sachet.
         length        OK   28/80
         banned        OK   ok
         must_include  OK   ok
  [PASS] protein-short
         output: 20g protein. Simple fuel.
         length        OK   25/80
         banned        OK   ok
         must_include  OK   ok
  [PASS] no-medical
         output: 20g protein in every sachet.
         length        OK   28/80
         banned        OK   ok
         must_include  OK   ok
  pass rate: 3/3 (100%)

=== skill_change ===
  [PASS] protein-sachet
         output: Fuel your day with protein.
         length        OK   27/80
         banned        OK   ok
         must_include  OK   ok
  [PASS] protein-short
         output: Protein, 20g a sachet.
         length        OK   22/80
         banned        OK   ok
         must_include  OK   ok
  [PASS] no-medical
         output: Fuel your day with protein.
         length        OK   27/80
         banned        OK   ok
         must_include  OK   ok
  pass rate: 3/3 (100%)

=== model_change ===
  [FAIL] protein-sachet
         output: Weight loss guaranteed!
         length        OK   23/80
         banned        X    guaranteed,weight loss
         must_include  X    missing ['protein']
  [FAIL] protein-short
         output: High protein, low fat, weight loss guaranteed.
         length        OK   46/80
         banned        X    guaranteed,weight loss,low fat
         must_include  OK   ok
  [FAIL] no-medical
         output: Cures fatigue. Weight loss guaranteed!
         length        OK   38/80
         banned        X    guaranteed,weight loss,cures
         must_include  X    missing ['protein']
  pass rate: 0/3 (0%)

=== suite ===
overall pass rate: 6/9 (67%)

=== same case, 5 runs (outputs vary) ===
  run 1: PASS  20g protein in every sachet.
  run 2: PASS  20g protein. No sugar added.
  run 3: PASS  20g protein in every sachet.
  run 4: FAIL  Weight loss guaranteed!
  run 5: PASS  20g protein in every sachet.
  pass rate: 4/5
  pass@5  (works at least once): PASS
  pass^5  (works every time):    FAIL

The prompt change and the skill-file change stay on-brief. The model swap invents claims. The 4/5 block is the other lesson. pass@5 is happy because it worked at least once. pass^5 is not, because it failed once. If this agent ships ads, you care about the second one.

In short

• You would not hire after one good answer. Do not ship an agent after one good demo.
• Golden examples are the work sample. The grader is the bar.
• Exact-match tests fit a ROAS report. They do not fit five different ad lines.
• Start with code checks. Add an LLM judge where wording can vary. Use humans to set and check the bar.
• Run each case more than once. “Works once” is pass@k. “Works every time” is pass^k.
• A bad ad is one bad ad. An agent that can write or delete needs evals for those actions, plus brakes.
• Twenty to fifty real cases, JSONL, CI, pass rate over time. That is enough to start.

Colour legend (retrieved chunks):

✓ relevant retrieved useful hit   ⚠ retrieved noise false alarm   ✗ relevant missed miss   · irrelevant not retrieved correct ignore

False alarms stay orange, not red, so you are not asked to tell red from green. Each cell also has a mark. The colour is a helper, not the only signal.

flowchart LR
  Q[Part-time parental leave?] --> R[Retrieve Top K chunks]
  R --> G[LLM writes the answer]
  G --> A[Reply to the employee]

The policy store

1. Top K — how many chunks go to the LLM

flowchart LR
  A[Score every chunk] --> B[Keep the top K]
  B --> C[Paste them into the prompt]

2. Recall@K — did we miss a fact?

Recall@K = relevant retrieved / relevant in the store

• Sentence: of the paragraphs that actually answer this question, how many did search bring back?
• This picture: 10 relevant paragraphs exist. K = 5 pulls 4 of them. 4 / 10 = 40%.
• What the other 6 are: ✗ misses. The duration cap, the return-to-work plan, partner leave. Still in the store. Not in the prompt.

3. Precision@K — how much of the prompt is noise?

Precision@K = useful retrieved / K chunks retrieved

• This picture: of 5, 2 are about normal vacation days or sick leave. 3 / 5 = 60%.
• What the 2 are: ⚠ retrieved noise. Close in wording (“leave”, “days”, “apply”). Wrong policy.

flowchart TB
  subgraph rec ["Recall@K — look at what is really relevant"]
    R["relevant retrieved / relevant in store"]
  end
  subgraph prec ["Precision@K — look at what we fetched"]
    P["useful retrieved / K"]
  end
  classDef tp fill:#d4f0e4,stroke:#1a9e55,color:#0d7a45
  classDef fn fill:#ddd6fe,stroke:#5b4cc4,color:#4c1d95
  class P tp
  class R fn

4. Top P — how freely the model picks the next word

5. Cost per request

K = 5 (example):

• Input tokens = 400 + 5 × 200 = 1,400
• Input cost = 1,400 / 1,000,000 × $0.50 = $0.00070
• Output cost = 250 / 1,000,000 × $1.50 = $0.000375
• Search = $0.002
• Total ≈ $0.0031 per request

K = 20, same question, search a bit heavier because more candidates are scored:

• Input tokens = 400 + 20 × 200 = 4,400
• Input cost = 4,400 / 1,000,000 × $0.50 = $0.00220
• Output cost = $0.000375 (same length answer)
• Search = $0.004
• Total ≈ $0.0066 per request, about 2.1× the K = 5 bill

The trade-off: raising K from 5 to 20

flowchart LR
  A[Raise K 5 to 20] --> B[Recall usually up]
  A --> C[Precision often down]
  A --> D[Input tokens up]
  D --> E[Cost up]
  D --> F[Latency up]

What I actually do

• Build a small labelled eval set. 30 to 50 real questions is enough to start. For each question, mark the chunk ids that should come back. Use the parental-leave style: one clear ask, a known gold set.
• Measure Recall@K and Precision@K across K. I usually try 1, 3, 5, 10, 20 on the same set. Plot both. Pick the K where recall is high enough and precision has not fallen off.
• Use a reranker. Retrieve 20 (cheap, wide). Keep the top 5 after a rerank model. You borrow the recall of K = 20 and aim for the precision of K = 5. The extra cost is the rerank pass, not 15 extra chunks in the LLM prompt.
• Keep chunk sizes sensible. I aim for about 200 to 500 tokens, with a little overlap. Tiny chunks lose the rule that sat in the next sentence. Huge chunks waste tokens and mix two policies in one blob.
• Keep top_p and temperature low for factual answers. HR, policy, finance. I do not want variety. I want the sentence that is already in the chunk.
• Track cost and p95 latency per request. Average latency hides the slow question. Cost per request tells you if last week’s K change actually moved the bill.

flowchart LR
  A[Retrieve 20] --> B[Rerank]
  B --> C[Keep top 5]
  C --> D[LLM]

A tiny check on the toy numbers

def recall_precision_at_k(relevant, retrieved):
    hits = sum(1 for chunk in retrieved if chunk in relevant)
    k = len(retrieved)
    recall = hits / len(relevant)
    precision = hits / k
    return hits, k, recall, precision


relevant = {
    "leave-pt-1",
    "leave-pt-2",
    "leave-pt-3",
    "leave-pt-4",
    "leave-pt-5",
    "leave-pt-6",
    "leave-pt-7",
    "leave-pt-8",
    "leave-pt-9",
    "leave-pt-10",
}

# Recall picture: 4 of 10 relevant, 1 vacation chunk.
hits, k, rec, prec = recall_precision_at_k(
    relevant,
    ["leave-pt-1", "leave-pt-2", "leave-pt-3", "leave-pt-4", "vacation-1"],
)
print(f"clean  Recall@{k} = {hits}/{len(relevant)} = {rec:.0%}")
print(f"clean  Precision@{k} = {hits}/{k} = {prec:.0%}")

# Precision picture: 3 useful, 2 noise.
hits, k, rec, prec = recall_precision_at_k(
    relevant,
    ["leave-pt-1", "vacation-1", "leave-pt-3", "sick-1", "leave-pt-7"],
)
print(f"noisy  Recall@{k} = {hits}/{len(relevant)} = {rec:.0%}")
print(f"noisy  Precision@{k} = {hits}/{k} = {prec:.0%}")

clean  Recall@5 = 4/10 = 40%
clean  Precision@5 = 4/5 = 80%
noisy  Recall@5 = 3/10 = 30%
noisy  Precision@5 = 3/5 = 60%

Cheat sheet

Here is a toy inbox. 10,000 mails. 100 of them are spam. The other 9,900 are normal.

A lazy model says “not spam” for every mail.

• It is right on 9,900 mails.
• Accuracy = 9,900 / 10,000 = 99%.
• It caught zero spam.
• Recall = 0 / 100 = 0%.

Accuracy looks like a score. When the thing you care about is rare, it can hide a model that never even tries.

The fix is two questions, not one. I keep them on four colours for the rest of this post. The colours do not change.

Colour legend (same chips in the text, the tables, the formulas, and the GIFs):

✓ TP correct catch   ⚠ FP false alarm   ✗ FN miss   · TN correct ignore

False alarms are orange, not red, so you are not asked to tell red from green. Each cell also has a mark. The colour is a helper, not the only signal.

flowchart LR
  A[The model made a call] --> B{Was it actually the thing?}
  B -->|yes, and we flagged it| TP["✓ TP correct catch"]
  B -->|no, but we flagged it| FP["⚠ FP false alarm"]
  B -->|yes, and we missed it| FN["✗ FN miss"]
  B -->|no, and we left it| TN["· TN correct ignore"]
  classDef tp fill:#d4f0e4,stroke:#1a9e55,color:#0d7a45
  classDef fp fill:#fde6d0,stroke:#d46a2e,color:#9a3b0a
  classDef fn fill:#ddd6fe,stroke:#5b4cc4,color:#4c1d95
  classDef tn fill:#e5e7eb,stroke:#6b7280,color:#4b5563
  class TP tp
  class FP fp
  class FN fn
  class TN tn

Four outcomes, in plain words

Every binary call lands in one box.

“Positive” here means the class we are hunting. Spam. Fraud. A customer who will leave. Not “good news”.

The confusion matrix, coloured

From here I use one small inbox so the arithmetic stays in your head.

100 mails. 20 are spam. 80 are normal. The model flags 16 as spam.

Check the margins. They have to add up.

• Real spam = 12 TP + 8 FN = 20
• Flagged spam = 12 TP + 4 FP = 16
• Real normal = 4 FP + 76 TN = 80
• Total = 12 + 4 + 8 + 76 = 100

Accuracy on this inbox is (12 + 76) / 100 = 88%. Fine as a headline. It still does not tell you how the 12 mistakes happened.

Layout note: this teaching grid puts TP top-left. scikit-learn’s confusion_matrix with labels=[0, 1] prints TN top-left. Same four numbers. Different corner. Read the labels before you quote a cell.

Precision: of everything we flagged, how much was right?

Precision looks at the predicted-positive column. Here that is “flagged as spam”.

Precision = TP / ( TP + FP )

• Sentence: of everything the model flagged, how much was right?
• Question: how many of the flagged items are relevant?
• This inbox: 12 / (12 + 4) = 12 / 16 = 75%

Four of the 16 flags were false alarms. Those are invoices sitting in junk, or customers you rang for no reason.

Recall: of everything that was really there, how much did we catch?

Recall looks at the actually-positive row. Here that is “really spam”.

Recall = TP / ( TP + FN )

• Sentence: of everything that was really there, how much did the model catch?
• Question: how many of the relevant items did we flag?
• This inbox: 12 / (12 + 8) = 12 / 20 = 60%

Eight spam mails reached the inbox. Those are the misses.

Recall is also called True Positive Rate (TPR) and sensitivity. Same fraction. Same row.

flowchart TB
  subgraph prec ["Precision — look at the model's Predictions"]
    P["TP / (TP + FP)<br/>the flagged column"]
  end
  subgraph rec ["Recall — look at what is Really there"]
    R["TP / (TP + FN)<br/>the actual-positive row"]
  end
  classDef tp fill:#d4f0e4,stroke:#1a9e55,color:#0d7a45
  classDef fn fill:#ddd6fe,stroke:#5b4cc4,color:#4c1d95
  class P tp
  class R fn

A memory trick that stays accurate

• Precision looks at the model’s Predictions. That is the flagged column.
• Recall looks at what is Really there. That is the actual-positive row.

Both start with TP. They divide by a different partner.

The fishing net

Think of catching fish in a lake.

The fish are the real positives — the spam, the fraud, the customers who will churn. Rocks look a bit like fish from far away.

A wide net scoops almost every fish. You miss very few. That is high recall. The net also picks up a pile of rocks. Those rocks are false alarms. Precision is lower.

A small net that you dip only where you are sure: almost everything in the net is a fish. Precision is high. Many fish are still swimming in the lake. Those are misses. Recall is lower.

You cannot usually have a net that is both very wide and very picky. The size of the net is your decision threshold.

On the GIF lake there are 8 fish and 6 rocks.

• Wide net: 6 fish + 4 rocks. Precision 6/10 = 60%. Recall 6/8 = 75%.
• Small net: 4 fish + 0 rocks. Precision 4/4 = 100%. Recall 4/8 = 50%.

A churn model, worked out

Same four boxes. Different business.

Last quarter you had 200 customers. 50 actually left. Your model flagged 40 as “will churn”.

Of those 40 flags, 32 did leave. 8 stayed. You missed 18 who left. You correctly left 142 stayers alone.

Margins:

• Real churn = 32 + 18 = 50
• Flags = 32 + 8 = 40
• Real stay = 8 + 142 = 150
• Total = 32 + 8 + 18 + 142 = 200

Now the two questions, slowly.

1. Precision = 32 / (32 + 8) = 32 / 40 = 80%. Of the people we rang with a retention offer, 80% were actually going to leave.
2. Recall = 32 / (32 + 18) = 32 / 50 = 64%. Of the people who actually left, we had flagged 64% in time.
3. Accuracy = (32 + 142) / 200 = 174 / 200 = 87%. Looks healthy. It still hides those 18 misses and 8 false alarms.

If a retention call costs money, the 8 false alarms are wasted calls. If a lost customer costs more, the 18 misses hurt more. That is the next section.

100% precision, 100% recall, and why you rarely get both

100% precision means FP = 0. Every flag was right. You can still miss a pile of real cases. A model that flags only one obvious spam mail, and gets it right, has perfect precision and terrible recall.

100% recall means FN = 0. You caught every real case. You can still flood people with false alarms. Flag every mail as spam and recall is 100%. Precision is 20 / 100 = 20% on the toy inbox.

Most models output a score, then you pick a cut. That cut is the net size.

Toy scores I made for the slider GIF. Four real positives at 0.94, 0.86, 0.78, 0.52. Four real negatives at 0.64, 0.40, 0.22, 0.08.

Same eight points. Three different nets.

flowchart LR
  A[Low threshold] --> B[High recall]
  A --> C[Lower precision]
  D[High threshold] --> E[High precision]
  D --> F[Lower recall]

When to care more about which

Pick the metric that matches the cost.

A ⚠ false alarm wastes time or hides a real message. A ✗ miss lets the bad thing through.

There is no universal winner. Write down the two costs. Then pick the net size.

F1, in one page

If someone asks for one number, F1 is the usual one. It is the harmonic mean of precision and recall:

F1 = 2 × Precision × Recall / (Precision + Recall)

On the 100-mail inbox: 2 × 0.75 × 0.60 / (0.75 + 0.60) = 0.90 / 1.35 = 0.667, which people write as 66.7% or 0.67.

Why harmonic, not the plain average?

The harmonic mean is pulled toward the smaller of the two numbers. If one of them is terrible, F1 is terrible.

Same 50% average. Model A almost never finds the thing you care about. F1 says that. The plain average does not.

On the churn toy: 2 × 0.80 × 0.64 / (0.80 + 0.64) = 1.024 / 1.44 = 0.711, or 71.1%.

False Positive Rate is the other axis people plot. FPR = FP / (FP + TN). On the 100-mail inbox that is 4 / (4 + 76) = 4 / 80 = 5%. It asks: of the normal mails, how many did we wrongly flag?

A ROC curve plots recall (TPR) against FPR as you slide the threshold. A precision-recall curve plots precision against recall on the same slide. When positives are rare, the PR curve is usually the clearer picture. That is the 99% accuracy trap in graph form.

A tiny scikit-learn check

Same 100 labels as the teaching matrix. 1 is spam. 0 is normal mail (ham in the report). I ran this. The printout below is the real output.

from sklearn.metrics import (
    confusion_matrix,
    precision_score,
    recall_score,
    f1_score,
    classification_report,
)

# 20 spam, 80 ham.
# Predictions: 12 TP, 4 FP, 8 FN, 76 TN.
y_true = [1] * 20 + [0] * 80
y_pred = [1] * 12 + [0] * 8 + [1] * 4 + [0] * 76

print(confusion_matrix(y_true, y_pred, labels=[0, 1]))
print("precision", precision_score(y_true, y_pred, pos_label=1))
print("recall", recall_score(y_true, y_pred, pos_label=1))
print("f1", f1_score(y_true, y_pred, pos_label=1))
print(classification_report(y_true, y_pred, target_names=["ham", "spam"]))

[[76  4]
 [ 8 12]]
precision 0.75
recall 0.6
f1 0.6666666666666666
              precision    recall  f1-score   support

         ham       0.90      0.95      0.93        80
        spam       0.75      0.60      0.67        20

    accuracy                           0.88       100
   macro avg       0.83      0.77      0.80       100
weighted avg       0.87      0.88      0.87       100

Read the sklearn matrix as:

Spam precision 0.75, spam recall 0.60, spam F1 0.67. Same fractions as the hand working.

Official docs: precision_score, recall_score, f1_score, confusion_matrix, classification_report.

Cheat sheet

100-mail toy: P = 75%, R = 60%, Acc = 88%, F1 = 66.7%. Churn toy: P = 80%, R = 64%, Acc = 87%, F1 = 71.1%.

In short

Accuracy can print 99% while the model never catches the rare thing you hired it for. Split the story. Precision asks, of the flags, how many were real. Recall asks, of the real cases, how many did we flag. I keep that as four colours: ✓ TP catch, ⚠ FP false alarm, ✗ FN miss, · TN correct ignore. Precision is the flagged column. Recall is the real-positive row. A wide net raises recall and lets rocks in. A small net raises precision and leaves fish in the lake. You pick the net from the cost of a false alarm versus the cost of a miss. F1 is the harmonic mean when you need one number. sklearn will print the same fractions if you give it the same labels.

It guesses the next token. It sticks that token on the end. Then it guesses again.

In this post the token is one character. The training text is a small dump of Shakespeare. The model is tiny. The shape is the same shape as GPT.

I am walking the same path as Andrej Karpathy’s lecture Let’s build GPT: from scratch, in code, spelled out. This is a concept post, not a line-by-line code dump. You can run the code later. Links are at the end.

flowchart LR
  A[Read past characters] --> B[Guess the next one]
  B --> C[Append the guess]
  C --> A

Want the code side? The Tiny GPT code walkthrough goes through the Colab notebook and gpt.py step by step. This page stays on the ideas.

What ChatGPT is doing

A language model is a next-token machine.

• You give it a sequence.
• It returns a list of probabilities for what comes next.
• You sample from that list (or pick the top item).
• You append the chosen token and repeat.

Same prompt can give a different answer. That is not a bug. The model is probabilistic. If “e” has 40% and “o” has 25%, sometimes you get “e”, sometimes “o”.

GPT means Generative Pre-trained Transformer.

ChatGPT sits on that idea, at huge scale, plus extra training. We are not rebuilding ChatGPT here. We are building the small educational cousin: a character-level model on about 1 MB of plays.

Karpathy’s earlier Zero to Hero videos (including makemore) already trained tiny character models with simpler nets. This lecture upgrades that idea to a Transformer.

Tiny Shakespeare is the playground

The dataset is tiny Shakespeare: a public dump of Shakespeare plays, roughly 1 million characters, about 1 MB.

You do not feed the whole book in one shot. You cut it into short windows and ask, again and again: given this window, what is the next character?

A trained toy model starts to babble in a Shakespeare-like way. You will see ROMEO:, thou, and line breaks. You will also see nonsense. That is expected. The data is small. The model is small. The tokenizer is one character at a time.

Think of it as a practice ground, not a product.

Tokens and the 65-character alphabet

Open the file. Collect every unique character. Sort them. In this dataset you get 65 symbols:

• letters (a–z, A–Z)
• digits
• space, newline
• punctuation (., ,, ?, !, :, and a few more)

Each character gets an integer id. Encode turns text into ids. Decode turns ids back into text. The exact tables are in the code walkthrough. The idea is simple: the network only sees integers.

Character tokens vs subword tokens

Real systems almost never use one token per character.

Trade-off:

• Small vocab → embedding table is small, but sequences get long.
• Large vocab → sequences get short, but the embedding table is huge.

We keep character-level so the story stays visible. You can watch T, then o, then space.

We never train on the whole play at once

Split the encoded file 90% train / 10% val. The val slice is for checking. Do not tune as if you had already seen it.

Two size knobs matter more than the rest.

A chunk of length T + 1 packs T training examples. Take 9 characters. You get 8 (context → next character) pairs.

Example with Hello!?? (8 context steps):

Why train on length 1, 2, 3, … up to block_size? Generation often starts from one token (sometimes just a newline). The model has to work with a short context, not only a full window.

The batch dim stacks many such chunks. They do not talk to each other. Shape to keep in your head: B × T predictions per step.

flowchart TB
  text[Tiny Shakespeare ids] --> split[90% train / 10% val]
  split --> window[Random window of length T+1]
  window --> x[Context x: first T]
  window --> y[Targets y: last T, shifted by 1]
  x --> gpu[Stack B windows]
  y --> gpu

A weak start: only look at the last character

The first model is a bigram.

Each token has a row in an embedding table. That row is used as the logits for “what comes next”. Token 47 does not look at token 18. Tokens do not talk.

• Optimizer: AdamW
• Loss: cross-entropy on the 65-way guess
• Random start: about (-\ln(1/65) \approx 4.17)
• After a short train in the lecture: loss in the ~2.5 region

Those loss numbers are from Karpathy’s educational run, not from a stopwatch on my machine.

Generation from a bigram is local. It can learn that q likes u, or that : often starts a new line. It cannot hold “we are inside a Romeo speech” across a sentence. There is no conversation between positions.

First, a matrix trick: average the past

Before Q, K, and V, the lecture builds a simpler idea.

For each position, take a weighted mix of the past (and the present). Do not look at the future.

A Python loop can do that. A lower-triangular matrix can do it in one multiply.

• Put 1s on and below the diagonal.
• Put 0s above it.
• Multiply. Position 3 only sees positions 0, 1, 2, 3.

Turn those 0/1 weights into a softmax, and you get a soft average. That is the skeleton of attention. The missing piece is: the weights should depend on content, not only on “is this in the past?”.

flowchart LR
  A[Past tokens] --> B[Lower-triangular mask]
  B --> C[Softmax weights]
  C --> D[Weighted sum]
  D --> E[One vector per position]

How tokens talk: query, key, value

Now each token builds three vectors from its embedding (plus its position, once we add those):

Attention, for one head:

1. Every token makes Q, K, and V.
2. Affinity of “me” to “you” is the dot product of my Q with your K.
3. Divide by (\sqrt{d}) so the dots do not explode before softmax. (d) is the head size.
4. Hide the future with a causal mask (next section).
5. Softmax turns scores into weights that sum to 1.
6. Mix the V vectors with those weights.

[ \mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{\top}}{\sqrt{d}}+M\right)V ]

(M) is 0 where looking is allowed, and (-\infty) on the future, so those weights become 0 after softmax.

The Linear layers, the (B, T, T) weight matrix, and the tril mask are in the code walkthrough.

flowchart TB
  tok[Token + position embedding] --> Q[Query]
  tok --> K[Key]
  tok --> V[Value]
  Q --> dots["Q · Kᵀ / √d"]
  K --> dots
  dots --> mask[Causal mask]
  mask --> sm[Softmax]
  sm --> mix[Weighted sum of V]
  V --> mix

If the current character is ? at the end of To be or not to be?, its Query can match Keys that look like a question setup. Those positions then donate their Values. The model is weighing how useful each past character is for this next guess.

Notes that save later confusion

• Attention is communication. Tokens send messages. You can draw it as a graph: a node per token, an edge when the weight is non-zero.
• Attention has no built-in sense of order. Swap the tokens and the dots still work. That is why we add position embeddings. “I am character 3” is extra information, added to the token embedding.
• Rows in a batch stay independent. Window A does not attend to window B.
• Self-attention: Q, K, V all come from the same sequence (the play so far).
• Cross-attention: Q comes from one sequence, K and V from another. The original translation Transformer uses this so the decoder can read the encoder.
• The (\sqrt{d}) scale is from the 2017 paper. Without it, softmax gets peaky and gradients get weak.

Do not peek at the future

If the target at position 3 is the character that actually sits at position 4, the model must not read position 4 while it predicts. That would be copying the answer.

So we use a causal (lower-triangular) mask.

For the five characters T o _ b e:

A group chat where you may only read messages that already arrived. You cannot read the ones that are still being typed.

This masked self-attention is why people call GPT a decoder-only Transformer. The 2017 encoder is allowed to look left and right. A GPT block is not.

Many heads, then a private think step

Multi-head attention is several of those Q/K/V channels in parallel.

• Head A might track “are we in a speaker name?”
• Head B might track “should a comma come next?”
• Head C might track “did a question start?”

Each head is smaller (n_embd / n_head). We concat the heads and project back to the model width.

After they talk, each token still needs to think on its own. That is a small MLP (feed-forward net) applied at every position, same weights everywhere:

1. Widen (in the lecture, to 4 × n_embd).
2. ReLU.
3. Project back.
4. Dropout.

Attention = communicate. MLP = compute. That pair is one block. Then you repeat the block.

flowchart TB
  x[x] --> ln1[LayerNorm]
  ln1 --> attn[Multi-head self-attention]
  attn --> add1["x = x + attn  (residual)"]
  add1 --> ln2[LayerNorm]
  ln2 --> mlp[Feed-forward MLP]
  mlp --> add2["x = x + mlp  (residual)"]

The lecture’s gpt.py uses pre-norm: LayerNorm, then the sub-layer, then add. The 2017 paper drew post-norm (add, then norm). Pre-norm is common now because it trains more calmly.

Residual roads and LayerNorm

Deep stacks need two boring, important tricks.

Residual (skip) connection. The block output is x + sublayer(x), not just sublayer(x). The original signal still has a highway through the stack. Gradients can travel that highway during training. Without it, a 6-layer toy already gets harder to train.

LayerNorm. For each token, rescale the vector so the numbers stay in a sane range. Combined with residuals, this keeps the hidden states from drifting.

Dropout. Randomly drop some activations in attention weights, projections, and the MLP. On a 1 MB dataset this is regularisation. The lecture’s bigger run uses dropout = 0.2.

The full GPT shape

Put the pieces in order.

1. Token embedding (65 rows, each of size n_embd).
2. Position embedding (block_size rows).
3. Add them.
4. Repeat n_layer blocks: masked multi-head attention, then MLP, each with residual + LayerNorm + dropout.
5. Final LayerNorm.
6. Linear head to 65 logits.
7. Softmax → sample → append → crop to the last block_size tokens → repeat.

flowchart TB
  ids[Character ids] --> te[Token embedding]
  ids --> pe[Position embedding]
  te --> add[Add]
  pe --> add
  add --> b1[Block 1]
  b1 --> b2[Block 2]
  b2 --> dots[…]
  dots --> bn[Block N]
  bn --> lnf[Final LayerNorm]
  lnf --> head[Linear to 65 logits]
  head --> sample[Softmax and sample]
  sample --> append[Append character]
  append --> crop[Keep last block_size]
  crop --> ids

This is a decoder-only Transformer.

The original 2017 Transformer was built for translation. It has two stacks:

GPT throws away the encoder and the cross-attention. There is only one stream: continue the text you already have. Chat-style models still look like this. Your prompt and the reply sit in the same sequence. The causal mask makes sure the model does not read the reply while it is still predicting it.

Scale up, still not ChatGPT

The lecture starts tiny (block_size = 8, a few channels) so you can print tensors and see the mask. Then it scales to a small-but-real toy. Numbers below are from that educational gpt.py run, not from us.

At 1.48 the samples look more like a play. You still would not stage them.

Finished, cleaned-up code lives in two places:

• Lecture notebook and scripts: karpathy/ng-video-lecture (bigram.py, gpt.py)
• The later, more serious repo: karpathy/nanoGPT

nanoGPT is the same idea pointed at GPT-2-scale training. It is still not ChatGPT.

What this tiny model skipped

Honest limit: this is a teaching model. It shows the skeleton. It does not write reliable essays, follow policies, or use tools. Keep the architecture in your head. Do not treat the samples as a real play, or as ChatGPT.

Try it yourself

These are labs, not social posts.

• Lecture video (full walkthrough, not only one timestamp): Let’s build GPT: from scratch, in code, spelled out
• Browser notebook: Colab — nanoGPT / lecture companion
• Code: github.com/karpathy/ng-video-lecture
• Cleaner follow-on: github.com/karpathy/nanoGPT
• The 2017 paper: Attention Is All You Need
• Subword tokenizers, when you leave character-level: tiktoken, SentencePiece
• Tiny Shakespeare source used in the lecture: the classic char-rnn input.txt

If you only have one hour: watch the video with the Colab open. Print one batch. Print the causal mask. Generate 200 characters from the bigram, then from the GPT. The gap is the point of the lecture.

For the line-by-line path through those files, use the Tiny GPT code walkthrough.

Glossary

Sources

• Let’s build GPT: from scratch, in code, spelled out — Andrej Karpathy
• Lecture Colab
• ng-video-lecture (bigram.py, gpt.py)
• nanoGPT
• Vaswani et al., Attention Is All You Need (2017)
• Zero to Hero / makemore (prior context)
• tiktoken

Loss figures, hyper-parameters, and the “about 15 minutes on an A100” note are from that lecture run. Treat them as teaching examples.

In short

A language model guesses the next character, then the next, then the next. Training is many windows of “past → next character” and a weight update (AdamW) that makes the right character more likely. A bigram cannot let those characters talk. Self-attention can: Query asks, Key labels, Value carries, a causal mask hides the future, and softmax mixes the past. Stack multi-head attention, a per-token MLP, residuals, LayerNorm, and dropout, and you have a decoder-only Transformer — the GPT shape — trained here on tiny Shakespeare. ChatGPT is that shape at a much larger scale, with subword tokens and extra training stages. This post is the small version you can actually hold in your head.

Want the tensors and the training loop? Open the Tiny GPT code walkthrough.

I wanted numbers from this machine, not a story from someone else’s laptop. So I started Postgres 16 in Docker, made a toy lunch_notes table, and ran a small Python load script. I counted commits per minute (RPM) for 12 seconds at a time.

Before the stopwatch: what a write actually does.

Foundation — what happens on a write

A row does not go straight to disk.

First it lands in shared buffers — RAM that the whole instance shares.

From there, two jobs start at the same time. They do not wait for each other.

Job 1: WAL

• A WAL record is written into the WAL buffer — still memory.
• Later that record is flushed to the WAL on disk.
• A helper called wal_writer wakes on a timer (wal_writer_delay, default 200 ms) and flushes what it can.
• COMMIT can also force a flush right now.

Job 2: the dirty page

• The table page in shared buffers is now dirty (it has a change that is not on disk yet).
• Later the background writer or the checkpointer copies that page into the table files.
• The background writer wakes every 200 ms (bgwriter_delay). It also wakes when shared buffers are getting full (buffer pressure).

The LSN gate

The two jobs can run at different speeds. The on-disk order is still strict.

Postgres stamps every dirty page with an LSN — the WAL position of the record that changed it.

How the gate works:

• A dirty page in shared buffers holds an LSN. That is the WAL position of the record that changed it.
• The WAL stream is a line of records with increasing LSNs: 10, then 20, then 30, then 42…
• Before that page may be written to the table file on disk, Postgres asks one question: has WAL been flushed at least past this page’s LSN?
• If the flush is still behind, the page waits at the gate. It does not sneak into the table files.
• When the WAL flush catches up past that LSN, the gate opens. Then the page may hit the table files.

Before the background writer or the checkpointer writes a page to the table files, it looks at the WAL flush position.

• If that LSN is already on disk in WAL: write the page.
• If it is not on disk yet: flush WAL first, then write the page.

You cannot write the table page to disk until the matching WAL record is already on disk. Crash recovery can replay WAL. It cannot invent a page that WAL never mentioned.

flowchart TB
  A[INSERT] --> B[Shared buffers]
  B --> C[WAL buffer]
  B --> D[Dirty table page + LSN stamp]
  C --> E["wal_writer 200ms or COMMIT"]
  E --> F[Disk WAL]
  D --> G{WAL flushed past this LSN?}
  G -->|not yet| E
  G -->|yes| H[bgwriter / checkpointer]
  H --> I[Table files]

Helpers at a glance

The wait knob

synchronous_commit only changes whether you wait for WAL to reach disk.

What off changes, and what it does not:

• The commit session no longer waits for WAL to reach disk.
• COMMIT returns once the WAL record is in the OS page cache (memory).
• This only affects the commit wait.
• The LSN gate between shared buffers and table files still applies exactly the same whether the knob is on or off. Dirty pages still cannot hit table files until WAL is flushed past their LSN.

Off is not “skip WAL.” WAL is still written. You just stop waiting. Think fire-and-forget, like UDP. If the machine crashes, the newest commits can vanish.

Why batching is cheaper

Each COMMIT can force a disk sync of WAL.

• 50 rows, 50 commits = 50 WAL flushes.
• 50 rows, one commit = one WAL flush.

Batching only groups WAL records. You get fewer, larger WAL disk writes — one COMMIT flush for many rows.

It does not batch how table pages (the heap) are written to disk. Those dirty pages still sit in shared buffers. The background writer still flushes them on its own schedule, one page at a time, and each of those writes still goes through the LSN gate.

The four runs below keep the durable wait on, then flip the knob, then put fifty rows in one commit.

What COMMIT waits for

You already saw shared buffers and the two disk paths. The four runs only flip one knob: does COMMIT wait for the WAL flush to disk?

flowchart TB
  A[My INSERT] --> B[Change sits in shared buffers]
  B --> C[WAL record is written]
  C --> D{synchronous_commit?}
  D -->|on| E[Wait for WAL flush to disk]
  D -->|off| F[Return now. Flush can happen later]

Off does not mean “no WAL.” WAL is still written. COMMIT just stops waiting for the disk flush. If the machine or the database crashes before the flush, you can lose the last commits. Crash recovery still works. The newest commits can still vanish.

Docker setup

Port 5432 was already busy, so I bound the container to 127.0.0.1:55432.

Disposable login only: user demo, password demo, database lunchbox.

sudo docker run -d --name intelliwise-pg-demo \
  -p 127.0.0.1:55432:5432 \
  -e POSTGRES_PASSWORD=demo \
  -e POSTGRES_USER=demo \
  -e POSTGRES_DB=lunchbox \
  postgres:16

The image reported PostgreSQL 16.15. Default synchronous_commit was on. WAL level was replica.

Four runs

Same toy table. Same ~12 second window. One change at a time. Shared VM, Python client — this is my loop, not a promise for your laptop.

1. Baseline — wait for every WAL flush

• synchronous_commit = on, 1 client, 1 row then COMMIT, 12.0 s
• 49,892 commits (~4,157.6 a second)
• Every 2.0 s the script printed a window. Those bounced 235,995.8 to 271,250.4 RPM. I report the full 12.0 s run.

2. Wait knob off — same pattern, only the knob

• SET synchronous_commit = off, 12.0 s
• 56,593 commits
• Small lift. Talking from Python on localhost and back was already a wait. The disk flush was not the only wait. Fast Docker storage also makes the flush cheaper than a slow laptop disk.
• If the OS or the database crashes before WAL flush, those last commits can disappear.

3. Batching — fifty rows, one commit

One row per commit pays the flush every time. Fifty rows in one commit pay it once.

• synchronous_commit = on, executemany 50 rows then one COMMIT, 12.001 s
• 14,560 commits, 728,000 rows
• Row RPM was about 14.59× the baseline commit RPM. I did not make the flush free. I paid it once per batch.

4. Four clients — wait still on, more writers

• synchronous_commit = on, 4 Python clients, each still 1 row then COMMIT, 12.014 s
• 79,064 commits together
• Four writers helped. They did not help like batching helped. Four loops still wait on each commit. Fifty rows in one commit skip most of those waits.

The 12-second card

RPM means “this many a minute,” guessed from a short wall-clock window. For batching, the headline is rows per minute. The others are commits per minute.

Stats after the ladder

I asked pg_stat_bgwriter and pg_stat_database on lunchbox. I also asked EXPLAIN on a toy INSERT — the plan, not a new stopwatch.

Those counters are the whole afternoon on that database, not one 12-second mode. Lots of hits. Almost no reads. Shared buffers already had the pages.

Toy SQL

Toy names. No real people. No production schema.

CREATE TABLE lunch_notes (
  id bigserial PRIMARY KEY,
  kid text NOT NULL,
  snack text NOT NULL,
  noted_at timestamptz NOT NULL DEFAULT now()
);

-- 1. Baseline: wait for WAL flush, one row
SET synchronous_commit = on;
BEGIN;
INSERT INTO lunch_notes (kid, snack) VALUES ('Sam', 'apple');
COMMIT;

-- 2. Wait knob off (can lose recent commits on crash before WAL flush)
SET synchronous_commit = off;
BEGIN;
INSERT INTO lunch_notes (kid, snack) VALUES ('Sam', 'apple');
COMMIT;

-- 3. Fifty rows, one commit (I used Python executemany)
SET synchronous_commit = on;
BEGIN;
INSERT INTO lunch_notes (kid, snack) VALUES
  ('Sam', 'apple'),
  ('Alex', 'pear');
  -- ...48 more rows...
COMMIT;

-- Plan only. Not a load test.
EXPLAIN
INSERT INTO lunch_notes (kid, snack) VALUES ('Sam', 'apple');

These snippets match the settings I used. They are not a second stopwatch.

Honest caveats

• One run. Shared 8-CPU / 8-vCPU VM. 15.6 GiB RAM. Other containers were up. Expect the next run to wiggle.
• Port 5432 was already in use. The demo port was 127.0.0.1:55432.
• The client is Python 3.13 with psycopg 3.3.6. Talking from Python is part of the wait. That is why sync-off only reached about 1.13×.
• Local Docker storage was fast. A slow laptop disk can make the WAL flush wait look much bigger.
• synchronous_commit = off can lose recent work if the OS or the database crashes before WAL flush. Official docs say this. I still used it as a teaching knob.
• Batching changed the unit. 3,639,835.0 is row RPM. Baseline 249,454.2 is commit RPM. Do not mix those two.
• RPM is guessed from 12-second windows printed every 2 seconds.
• Demo login is disposable. demo / demo / lunchbox. Not a real password. Not a public server.
• I am not promising these RPM numbers on your machine.

Terms used in this post

You do not need this table to follow the runs. It is only a quick lookup.

Official references (Postgres docs only):

• Write-Ahead Logging (WAL)
• WAL internals
• synchronous_commit
• wal_writer_delay
• bgwriter_delay
• WAL reliability
• EXPLAIN
• The Statistics Collector

In short

A write lands in shared buffers. Two jobs start at once: a WAL record, and a dirty page. The LSN gate will not write the page to the table files until that WAL position is on disk. That gate is the same with the wait knob on or off. COMMIT’s wait knob is only the WAL flush. Batching only groups WAL records, not table-page writes. On my Docker run, turning the wait off was a small lift (~1.13×). Putting 50 rows in one commit was the big climb (~3.64M row RPM). Four clients helped (~1.58×). These are my 12-second numbers on a shared VM — not a promise.

The usual list is a checklist. This post is a lunch-box map.

Same 14 names. Grouped so you can see why each box exists, and which box to open first.

I did not invent these papers. I only sorted the tray.

Each paper has three layers:

1. This map — why the box exists, a short Quick easy read, and the diagrams.
2. A longer guide — still brief, enough that you do not need to grind the PDF. Linked from the table and from each paper. These guides live under this post. They are not listed on the Writing index or in the RSS feed.
3. The real paper — official abstract, PDF, conference page, and a Dataset link only when a public corpus is honestly tied to that paper.

Skim the map. Open the longer guide when you want more detail. Open the paper when you want the proof.

flowchart TB
  A[Engine: Transformers] --> B[Place words: RoPE]
  A --> C[Read both ways: BERT]
  A --> D[See pictures: ViT]
  A --> E[Open recipe: LLaMA]
  E --> F[Follow instructions: InstructGPT + RLHF]
  E --> G[Cheap fine-tune: PEFT + LoRA]
  E --> H[Sparse giants: MoE]
  E --> I[Look stuff up: RAG]
  J[Old image makers: GAN + VAE] --> K[New image makers: Diffusion]

The habit to keep:

Do not memorise 14 titles. Learn the lunch boxes.

The 14 names (same order as the boxes below)

Names jump to the box. Paper goes to the official abstract. Longer guide is the unlisted sub-page.

Now the boxes.

Box 1 — The engine room

Attention Is All You Need (Transformers)

Longer guide → more detail without grinding the PDF

Before this, models often read words one after another, like a single-file line.

Attention lets every word look at every other word and ask: “Who matters for me right now?”

That paper (2017) is why ChatGPT-style models exist.

flowchart LR
  W1[word] --> A[Attention]
  W2[word] --> A
  W3[word] --> A
  A --> O[rich meaning]

Read when: you want the root of modern LLMs.

Quick easy read

What it is

• A new engine for reading sequences. No slow “one word, then the next” loop as the main trick.
• Every token can look at every other token in one go (self-attention).
• Stack those looks with a little feed-forward maths: that stack is a Transformer.
• The 2017 paper tested it on translation. The same engine later showed up in chat, code, and pictures.

Why people care

• GPUs like work they can do in parallel. Attention is much more parallel than old recurrent nets.
• One engine, many jobs: text, code, images (see ViT), even tools that call other tools.
• If you only open one paper on this tray, open this one.

Simple picture

Think of a meeting where everyone can look at everyone else at the same time, instead of passing a note down a single line. The word “bank” looks at “river” and “money”, then decides which neighbours matter.

Links: Paper (arXiv) · NeurIPS 2017 · Dataset: WMT 2014 translation task (the paper’s English–German / English–French news tests)

Box 2 — Where am I in the line?

RoPE (Rotary Position Embedding)

Longer guide → more detail without grinding the PDF

Transformers need to know order. “Dog bites man” is not “Man bites dog.”

RoPE rotates number vectors so position is baked into attention. Many open models, including LLaMA-style stacks, use ideas like this.

In one line: the model needs a seat number, not just the word itself.

Quick easy read

What it is

• Attention is good at “who talks to whom.” It does not know where unless you tell it.
• RoPE (from the RoFormer paper) turns each token’s numbers a little, like rotating a clock hand, based on position.
• Nearby positions rotate a little. Far positions rotate a lot. Relative distance comes out of the rotation.

Why people care

• Order is not optional in language, logs, or code.
• A lot of later open LLMs (LLaMA-family and cousins) use rotary-style position.
• It stretches to longer sequences more kindly than some older “add a position vector” tricks.

Simple picture

Two people on a spinning ride. How much one has turned relative to the other is how far apart they sit. You do not need a separate name tag that says “seat 7.”

Links: Paper — RoFormer / Su et al. (arXiv)

Box 3 — Reading both ways

BERT

Longer guide → more detail without grinding the PDF

BERT is an encoder transformer. It looks left and right.

It is good at understanding: search, classify, fill blanks. It is not the usual “write an essay” model by itself.

In one line: read the whole sentence before you judge one word.

Quick easy read

What it is

• A Transformer that reads both ways (bidirectional), then gets fine-tuned for a job.
• Pre-training games: hide a word and guess it; sometimes guess if sentence B really follows sentence A.
• After that, a thin extra head can do search ranking, sentiment, span answers, and similar “understand this text” work.

Why people care

• For years this was the default way to make text features for products: classify a ticket, match a query, tag a span.
• It showed that one pretrained encoder plus a little extra training beats many custom architectures.
• Chat models that write usually use decoder stacks. BERT is the “read the ticket first” cousin.

Simple picture

Someone covers one word in a sentence and asks you to guess it — but you already saw the words on both sides. That is the fill-in-the-blank game BERT practised at huge scale.

Links: Paper (ACL Anthology) · arXiv · Pretraining corpus: English Wikipedia dumps (BERT used text passages, not lists/tables) · BooksCorpus source paper (Zhu et al., 2015) (the original BookCorpus dump is not a simple public download today)

Box 4 — Eyes for transformers

ViT (Vision Transformer)

Longer guide → more detail without grinding the PDF

Chop a picture into little patches, like stamp tiles. Treat each tile like a “word.” Run a transformer.

In one line: a photo becomes a strip of tiles, then attention reads the strip.

Quick easy read

What it is

• Same Transformer engine as text, pointed at images.
• Cut the photo into a grid of squares (often 16×16 pixels). Each square is a token.
• Train big, then transfer to smaller picture jobs. The paper’s headline result is image classification at scale.

Why people care

• You do not need a special “vision-only” backbone if you have enough data and compute.
• The same attention story now covers pixels and words — handy when a product mixes both.
• It is the clean “eyes” box on this tray: one idea, not a zoo of convolution tricks.

Simple picture

Take a poster, cut it into postage stamps, line the stamps up like a sentence, and let attention read the sentence. A cat’s ear tile looks at the whisker tiles and decides “cat.”

Links: Paper — Dosovitskiy et al. (arXiv) · Dataset: ImageNet (ViT pretrains on large labelled photo sets such as ImageNet-21k, then transfers to ImageNet-style tests; access is for research, not a random public zip)

Box 5 — An open language recipe

LLaMA

Longer guide → more detail without grinding the PDF

LLaMA (Meta) shared a strong large language model recipe with the research world. Many later open models follow the same shape.

In one line: a published cookbook for large language models, not a magic spell.

Quick easy read

What it is

• A family of decoder Transformers (7B to 65B in the first paper) trained on a lot of public text.
• Meta showed you can get strong results without a secret private corpus as the whole story.
• The recipe leaked into a thousand later models: same-shaped stacks, same “open weights + paper” habit.

Why people care

• Engineers can read a real training recipe, not only an API card.
• Smaller LLaMA-13B beating much bigger GPT-3 on many tests was the “wait, data and tokens matter” moment.
• If you ship on open weights, you will meet this family tree.

Simple picture

Not a magic spell. A published cookbook: flour, water, oven time. Other kitchens copy the loaf shape and change the toppings.

Links: Paper — Touvron et al. (arXiv) · Code / model card (Meta Llama GitHub) · Hugging Face LLaMA docs

Box 6 — Teach it to listen

A raw LLM can ramble. People want it to follow instructions and be useful. These two papers sit in one box.

flowchart LR
  P[Prompt] --> M[Model draft]
  M --> H[Human preference]
  H --> M2[Better model]

Read when: you care why chatbots feel polite and useful.

InstructGPT

Longer guide → more detail without grinding the PDF

InstructGPT: train / tune so the model follows prompts better. The OpenAI paper that named the chat-style habit: supervised demos, then preference training.

Quick easy read

What it is

• Start from a big pretrained model (here: GPT-3).
• Step 1: humans write good answers to prompts. Train on those (supervised fine-tune).
• Step 2: humans rank model answers. Train a reward model, then move the policy toward the ranks (RLHF).
• The 1.3B InstructGPT was often preferred to 175B GPT-3 on their prompt set — size is not the same as “does what I asked.”

Why people care

• This is why “write a polite email” works better than raw next-token dump.
• In products you meet this as alignment, safety filters, and “the model got nicer after we collected ranks.”
• Later chat models reuse the same three-step skeleton, with extra tricks.

Simple picture

A junior first copies marked homework (demos). Then a reviewer points at two answers: “this one, not that one.” The junior practises toward the pointing finger, not toward a written rulebook.

Links: Paper — Ouyang et al. (arXiv)

RLHF

Longer guide → more detail without grinding the PDF

RLHF: humans (or preference data) score answers; the model moves toward better preference.

InstructGPT uses RLHF. The older landmark is Christiano et al.: teach an agent from human preference instead of a hand-written reward.

Quick easy read

What it is

• You often cannot write a perfect score function for “be helpful” or “do a backflip nicely.”
• Show a human two clips (or two answers). They pick a winner.
• Fit a reward model to those picks. Train the agent with RL on that reward.
• InstructGPT is the language-model version of this loop.

Why people care

• Taste, tone, and “don’t be rude” are preference problems, not unit tests.
• This is the named method behind a lot of chat fine-tunes — and behind a lot of later debate about whose taste got encoded.
• If you only remember one sentence: humans compare, the model climbs.

Simple picture

Two robot dance videos. You tap the less-awkward one. You never write the physics of dancing. The robot still gets better at dancing.

Links: Paper — Christiano et al. (arXiv) · Language-model version: InstructGPT

Box 7 — Cheap fine-tuning

Full fine-tuning touches all the weights. That is expensive.

PEFT is the family name. LoRA is the small-update trick most teams actually ship.

LoRA

Longer guide → more detail without grinding the PDF

LoRA = freeze the big model; train tiny low-rank add-ons.

In one line: keep the whole textbook; only rewrite a sticky note.

Quick easy read

What it is

• Leave the pretrained weights frozen.
• Beside some big matrices (often attention projections), train two thin matrices whose product is a small “update.”
• At serve time you can merge that update back in, so you do not pay extra latency like some older adapter stacks.

Why people care

• One base checkpoint, many cheap specialist patches (support tone, internal jargon, a tool habit).
• Disk and GPU memory stay sane. You are not storing 13 copies of a 13B model.
• This is the default “we fine-tuned it” in a lot of real stacks.

Simple picture

A huge printed textbook stays on the shelf. You teach a new course by sticking a few notes on the hard pages. Swap the notes; the book stays.

Links: Paper — Hu et al. (arXiv)

PEFT

Longer guide → more detail without grinding the PDF

PEFT = Parameter-Efficient Fine-Tuning. Not one paper. A family: LoRA, adapters, prompt/prefix tuning, and friends.

Use LoRA unless you have a reason not to. Treat the rest as “know why it exists.”

Quick easy read

What it is

• Same job as full fine-tune: specialise a strong base model.
• Change only a small set of extra numbers (or a small add-on), not every weight.
• Adapters (Houlsby et al.) were an early member: tiny bottleneck layers slid into a frozen BERT. LoRA later became the popular low-rank version.
• Hugging Face PEFT is the library most engineers actually import.

Why people care

• Full copies of a 70B model per customer task do not fit a budget.
• You can keep one frozen backbone and hot-swap skills, like plugins.
• If someone says “we did PEFT,” ask which method. Often they mean LoRA.

Simple picture

The library building stays put. Each new subject gets a thin pamphlet, not a second building.

Links: Hugging Face PEFT docs · LoRA paper · Adapter paper — Houlsby et al. (arXiv) · PMLR / ICML version · Google Research page

Box 8 — Sparse giants

MoE (Mixture of Experts)

Longer guide → more detail without grinding the PDF

Instead of one huge network always on, keep many expert networks. For each token, wake only a few.

In one line: many specialists; only the right ones stand up.

This box’s primary paper is Switch Transformers (Fedus, Zoph, Shazeer): a Transformer MoE that routes each token to one expert. The older root is Shazeer et al. 2017 (sparsely-gated MoE).

Quick easy read

What it is

• A dense model uses all of its weights for every token. An MoE keeps a roster of expert feed-forward blocks.
• A small router picks which expert(s) see this token.
• Switch: pick one expert (simpler, cheaper communication than older top-k MoE).
• You can grow parameter count a lot without growing compute per token as fast.

Why people care

• This is how some “trillion parameter” language models stay runnable: most experts sleep.
• Same bill of GPU-time, more capacity — if routing stays healthy (load balance is the engineering tax).
• If a vendor says “sparse model,” they often mean this lunch box.

Simple picture

A hospital with many specialists. The triage nurse sends you to one doctor. The other doctors keep working on other patients. You do not call the whole staff for a sore throat.

Links: Primary paper — Switch Transformers (arXiv) · Lineage — Shazeer et al., sparsely-gated MoE (arXiv) · C4 corpus (defined in the T5 paper) (Switch pretrains on the Colossal Clean Crawled Corpus)

Box 9 — Look it up first

RAG (Retrieval-Augmented Generation)

Longer guide → more detail without grinding the PDF

Models forget and hallucinate. RAG fetches fresh notes from your files / search, then asks the model to answer with those notes.

In one line: open the binder before you answer.

Quick easy read

What it is

• Two memories: the weights (what the model already “knows”) and a search index (notes you can update).
• At question time: retrieve a few passages, then generate an answer conditioned on those passages.
• Lewis et al. used a seq2seq generator plus a dense index of Wikipedia.
• Today’s product RAG is the same sandwich: retrieve → stuff into the prompt → generate. The 2020 paper is the named origin.

Why people care

• You can fix facts by editing documents, not by retraining a 70B model.
• Provenance: you can point at the paragraph you used.
• This is the usual engineering answer to “the model made up a policy.”

Simple picture

Open-book test. You grab three photocopies from the binder, then write. Closed-book is hoping you memorised the binder.

Links: Paper — Lewis et al. (arXiv) · Retrieval corpus: Wikipedia dumps (the paper indexes Wikipedia passages)

Box 10 — Picture makers (classic)

Two older ways to make pictures. Diffusion (next box) mostly won the “pretty image from a prompt” race. These two still matter for the maths and for latent spaces.

GANs

Longer guide → more detail without grinding the PDF

GAN: a forger and a detective. The forger gets better until fakes fool the detective.

Quick easy read

What it is

• Two nets train together: a generator makes fakes from noise; a discriminator says real vs fake.
• The generator’s job is to fool the discriminator. That fight is the training signal.
• Goodfellow et al. (2014) is the landmark. Later years were a zoo of GAN variants.

Why people care

• First time many people saw sharp generated photos, not blurry averages.
• The “two players” idea shows up in other training tricks (including parts of modern image stacks).
• History box: know it so you know what diffusion replaced in products.

Simple picture

Art forger vs museum detective. Every week the forger studies what got caught. The paintings get scarily good. Nobody wrote “draw a cat” as a loss; the fight was enough.

Links: Paper — Goodfellow et al. (arXiv) · Dataset used in the paper: CIFAR-10

VAE

Longer guide → more detail without grinding the PDF

VAE: squeeze an image into a small code, then rebuild. Good for smooth “latents.”

In one line: art contest (GAN) vs zip-and-unzip (VAE).

Quick easy read

What it is

• An encoder maps a picture to a small cloud of numbers (a latent).
• A decoder rebuilds the picture from a sample of that cloud.
• The training deal: rebuild well, but keep the cloud simple (usually “look like a standard bell curve”).
• Kingma & Welling (2013/14) is the landmark variational autoencoder.

Why people care

• Latent spaces you can walk through: morph one face toward another.
• Modern image generators still compress with a VAE-like autoencoder, then do the fancy work in that smaller space (see Diffusion).
• If someone says “latent,” this is the zip-file intuition.

Simple picture

Stuff a poster into a tiny envelope of numbers, then unpack it. Nearby envelopes unpack into similar posters. That neighbourhood is the latent space.

Links: Paper — Kingma & Welling (arXiv)

Box 11 — Picture makers (now)

Diffusion (Stable Diffusion lineage)

Longer guide → more detail without grinding the PDF

Start with TV static. Step by step, remove noise until a clean picture appears. Modern image models lean hard on this idea (often with a VAE latent space).

In one line: messy chalk → wipe carefully → drawing.

Quick easy read

What it is

• DDPM (Ho, Jain, Abbeel): add noise to real photos until they are static; train a net to undo one noise step. Repeat the undo until you have a picture.
• Latent Diffusion (Rombach et al.): do that denoising in a small latent grid from an autoencoder, not on every pixel. That is the Stable Diffusion lineage this box names.
• Text can steer the undo via cross-attention (“a red bicycle in the rain”).

Why people care

• This is how most “type words, get a picture” tools work now.
• Latent diffusion made high-res generation cheap enough to run outside a giant lab.
• Same “iterative cleanup” idea later jumped to audio and video.

Simple picture

A dusty chalkboard. You do not draw the cat in one stroke. You wipe a little dust, then a little more, until the cat is there. Start from a blank dusty board (pure noise) and wipe toward the prompt.

Links: DDPM — Ho et al. (arXiv) · Latent Diffusion / Stable Diffusion lineage — Rombach et al. (arXiv) · LAION-400M (text-to-image training in the LDM paper) · LAION-400M project page · MS-COCO (eval set in that paper)

Suggested reading order (busy people)

1. Attention Is All You Need — the engine
2. BERT or skim a LLaMA overview — understand vs generate
3. RoPE (short) — position
4. InstructGPT + RLHF — why chat feels aligned
5. LoRA / PEFT — how you customise cheaply
6. RAG — how products stay factual
7. MoE — how giants stay fast enough
8. ViT — same engine, eyes
9. VAE → Diffusion (and peek at GANs for history)

You do not need all 14 in one weekend.

Honest caveats

• A common list is a map, not a PhD.
• Paper titles evolve (RAG stacks, diffusion variants, LLaMA versions). Learn the idea, then check the latest version.
• Dataset links are only where a public corpus is clearly named in the landmark paper. Some famous training sets (InstructGPT ranks, original BookCorpus dumps) are not a clean public zip.
• More lists may exist — this post only teaches these fourteen.

Sources

Official abstracts and resources used in this post (prefer these over roundups):

1. Vaswani et al., Attention Is All You Need
2. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding
3. Devlin et al., BERT (arXiv); Wikipedia dumps; BooksCorpus via Zhu et al.
4. Dosovitskiy et al., An Image is Worth 16×16 Words; ImageNet
5. Touvron et al., LLaMA: Open and Efficient Foundation Language Models; Meta Llama GitHub
6. Ouyang et al., Training language models to follow instructions with human feedback
7. Christiano et al., Deep Reinforcement Learning from Human Preferences
8. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models
9. Hugging Face PEFT; Houlsby et al., Parameter-Efficient Transfer Learning for NLP
10. Fedus, Zoph & Shazeer, Switch Transformers; Shazeer et al., Sparsely-Gated Mixture-of-Experts
11. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
12. Goodfellow et al., Generative Adversarial Nets
13. Kingma & Welling, Auto-Encoding Variational Bayes
14. Ho et al., Denoising Diffusion Probabilistic Models; Rombach et al., Latent Diffusion Models

Takeaway

Fourteen famous papers sit in a handful of lunch boxes: the transformer engine, position, deep reading, vision tiles, open LLM recipes, instruction plus human preference, cheap adapters, sparse experts, retrieval, and three ways to make pictures. Learn the box on this map, open the longer guide if you want more detail, then open the paper.