Diagnosing a model that isn't working¶
Every other tutorial shows you something that works. This one breaks things on purpose, so that when your own model misbehaves you have a procedure instead of a hunch.
The companion page When it fails lists six real failures from this project's history and what actually caused each one. Reading that page tells you the answers. This notebook makes you watch the checks fire.
The order matters¶
Work top to bottom. Each question is cheaper to answer than the one below it, and a yes at any level makes everything below it meaningless. There is no point tuning a learning rate on a corpus whose two sides do not match.
| # | Question | Cost to check |
|---|---|---|
| 1 | Is the data what you think it is? | seconds |
| 2 | Is the model learning anything? | one backward pass |
| 3 | Is it learning the wrong thing? | one training run |
| 4 | Is the measurement lying? | seconds |
| 5 | Is it the environment? | seconds |
Every check below returns PASS or FAIL against a stated threshold, and every one is short enough to copy into your own project.
Setup: a toy corpus we can break¶
Real translation needs hours of training, which makes it a poor teaching instrument: you cannot tell a bug from an undertrained model. So we use a synthetic English/Spanish task that a small Transformer solves in under a minute. When it doesn't solve it, we know we broke something.
Sentences look like Maria sees 42 old cats / Maria ve 42 viejos gatos. Two
properties matter:
- Word-for-word and same length, so a correct corpus has a length correlation of exactly 1.0.
- Names and numbers pass through untranslated, giving the alignment checks
anchors to look for.
Mariaand42must appear on both sides.
Nothing here is downloaded, so this notebook runs anywhere.
import math
import random
import warnings
from functools import partial
import pandas as pd
import torch
from torch import nn, optim
from torch.utils.data import DataLoader, Dataset
from torchlingo.config import Config
from torchlingo.data_processing.batching import collate_fn
from torchlingo.evaluation import compute_bleu
from torchlingo.inference import greedy_decode
from torchlingo.models import SimpleTransformer
from torchlingo.preprocessing.alignment import diagnose_alignment, shuffle_target_side
from torchlingo.training import train_model
warnings.filterwarnings("ignore")
NAMES = ["Maria", "Juan", "Ana", "Pedro", "Elena", "Carmen", "Diego", "Rosa"]
VERB = {"sees": "ve", "wants": "quiere", "finds": "encuentra", "likes": "aprecia"}
NOUN = {"cats": "gatos", "dogs": "perros", "birds": "pajaros", "fish": "peces"}
ADJ = {"old": "viejos", "young": "jovenes", "big": "grandes", "small": "pequenos"}
def make_corpus(n: int, seed: int) -> pd.DataFrame:
"""Build n parallel sentence pairs. Length varies so correlation is measurable."""
rng = random.Random(seed)
rows = []
for _ in range(n):
name = rng.choice(NAMES)
verb = rng.choice(list(VERB))
number = rng.randint(10, 99)
adjs = rng.sample(list(ADJ), rng.randint(0, 2))
noun = rng.choice(list(NOUN))
source = [name, verb, str(number), *adjs, noun]
target = [name, VERB[verb], str(number), *[ADJ[a] for a in adjs], NOUN[noun]]
rows.append((" ".join(source), " ".join(target)))
return pd.DataFrame(rows, columns=["src", "tgt"])
train_df = make_corpus(3000, seed=0)
val_df = make_corpus(400, seed=99)
train_df.head(3)
class Vocab:
"""A word-level vocabulary over both languages at once."""
def __init__(self, frames):
tokens = set()
for frame in frames:
for column in ("src", "tgt"):
for sentence in frame[column]:
tokens.update(sentence.split())
self.itos = ["<pad>", "<sos>", "<eos>"] + sorted(tokens)
self.stoi = {t: i for i, t in enumerate(self.itos)}
self.pad_idx, self.sos_idx, self.eos_idx = 0, 1, 2
def __len__(self):
return len(self.itos)
def encode(self, text):
return [self.sos_idx] + [self.stoi[w] for w in text.split()] + [self.eos_idx]
def decode(self, ids):
return " ".join(self.itos[i] for i in ids if i > self.eos_idx)
class Pairs(Dataset):
def __init__(self, frame, vocab):
self.frame = frame.reset_index(drop=True)
self.vocab = vocab
def __len__(self):
return len(self.frame)
def __getitem__(self, i):
row = self.frame.iloc[i]
return (
torch.tensor(self.vocab.encode(row.src)),
torch.tensor(self.vocab.encode(row.tgt)),
)
vocab = Vocab([train_df, val_df])
# label_smoothing is off so that 0.0 means "perfect" and ln(V) means "guessing".
# With the default 0.1 a converged model still sits near 0.6 on this vocabulary,
# which makes both reference points harder to reason about.
CFG = Config(batch_size=64, label_smoothing=0.0, pad_idx=0, sos_idx=1, eos_idx=2)
UNIFORM_LOSS = math.log(len(vocab))
def loader(frame, batch_size=64, shuffle=True):
return DataLoader(
Pairs(frame, vocab),
batch_size=batch_size,
shuffle=shuffle,
collate_fn=partial(collate_fn, pad_idx=vocab.pad_idx),
)
def new_model(seed=0):
torch.manual_seed(seed)
return SimpleTransformer(
src_vocab_size=len(vocab),
tgt_vocab_size=len(vocab),
d_model=128,
n_heads=4,
num_encoder_layers=2,
num_decoder_layers=2,
d_ff=256,
dropout=0.1,
)
def translate(model, sentences, max_len=12):
model.eval()
ids = [vocab.encode(s) for s in sentences]
width = max(len(x) for x in ids)
src = torch.tensor([x + [vocab.pad_idx] * (width - len(x)) for x in ids])
return [vocab.decode(o) for o in greedy_decode(model, src, max_len=max_len, config=CFG)]
def report(name, ok, detail):
"""Every diagnostic in this notebook prints through here."""
print(f"[{'PASS' if ok else 'FAIL'}] {name}\n {detail}")
return ok
print(f"vocabulary: {len(vocab)} types")
print(f"a model that has learned nothing scores ln(V) = {UNIFORM_LOSS:.3f}")
The control: a model that works¶
Before breaking anything we need to know what healthy looks like on this task. Keep these two numbers — every failure below is measured against them.
This takes about a minute.
healthy = new_model()
healthy_run = train_model(
healthy,
loader(train_df),
loader(val_df, shuffle=False),
num_epochs=20,
config=CFG,
optimizer=optim.Adam(healthy.parameters(), lr=1e-3),
)
HEALTHY_TRAIN = healthy_run.train_losses[-1]
HEALTHY_VAL = healthy_run.val_losses[-1]
print(f"\ntrain {HEALTHY_TRAIN:.3f} val {HEALTHY_VAL:.3f} (guessing would be {UNIFORM_LOSS:.3f})")
for source, output in zip(val_df.src[:3], translate(healthy, list(val_df.src[:3]))):
print(f" {source:32} -> {output}")
Question 1 — Is the data what you think it is?¶
This is first because it is the cheapest to check and the most expensive to miss. A corpus whose two sides drifted out of alignment still loads, still batches, still trains, and still produces a falling loss curve. Nothing raises.
This is not hypothetical. TorchLingo's own data/example.tsv shipped misaligned
for a period, and shuffle_target_side exists to reconstruct exactly that
failure on any corpus so you can watch the checks catch it.
The diagnostic is diagnose_alignment, which runs two independent checks:
- Length correlation. Translations preserve length approximately. Genuinely parallel text correlates near 0.97; unrelated pairs near 0.0.
- Anchor agreement. Names and numbers survive translation. The fraction of rows whose two sides share at least one should be high.
looks_aligned() requires correlation >= 0.80 and agreement >= 0.25.
def check_alignment(frame, label):
result = diagnose_alignment(frame)
return report(
f"corpus alignment ({label})",
result.looks_aligned(),
f"length_correlation={result.length_correlation:.3f} (need >=0.80) "
f"anchor_agreement={result.anchor_agreement:.3f} (need >=0.25)",
)
broken_df = shuffle_target_side(train_df)
check_alignment(train_df, "as built")
check_alignment(broken_df, "after shuffle_target_side")
print("\nthe same row, both ways:")
print(f" intact : {train_df.src[0]!r} -> {train_df.tgt[0]!r}")
print(f" broken : {broken_df.src[0]!r} -> {broken_df.tgt[0]!r}")
What it costs to skip this check¶
Train on the broken corpus and nothing announces a problem — the loss falls and the curve looks like training.
Give it the same 20 epochs the healthy model got, changing exactly one variable (the corpus), and the difference is unmistakable.
on_broken = new_model()
broken_run = train_model(
on_broken,
loader(broken_df),
loader(val_df, shuffle=False),
num_epochs=20,
config=CFG,
optimizer=optim.Adam(on_broken.parameters(), lr=1e-3),
)
print(f"\nlast 5 epochs on the broken corpus: {[round(x, 3) for x in broken_run.train_losses[-5:]]}")
print(f" it has flattened out at train {broken_run.train_losses[-1]:.3f}")
print(f" healthy, same 20 epochs: train {HEALTHY_TRAIN:.3f}")
print("\nwhat it produces:")
for source, output in zip(val_df.src[:3], translate(on_broken, list(val_df.src[:3]))):
print(f" {source:32} -> {output}")
Recognize it: the loss falls, then flattens out far above where a comparable run reaches — and the outputs are built from target-language vocabulary that has little to do with the input. Names and numbers, which a working model copies straight through, come out wrong or missing entirely.
That is the signature of a model that learned the target language and could not learn the mapping, because in this corpus there isn't one to learn.
Fix it: run diagnose_alignment on every corpus before training, and treat
looks_aligned() == False as a hard stop. Anything landing between the
thresholds is worth reading by hand. Realigning a corpus that has drifted is its
own job — see gale_church_align in torchlingo.preprocessing.alignment.
Question 2 — Is the model learning anything?¶
There is a specific number that means "learned nothing": the loss of a model that
guesses uniformly over the vocabulary, ln(V). A loss sitting there, flat, is
not a slow model — it is a disconnected one.
Here we break it the most common way there is: a learning rate of zero.
!!! note "Why the flat loss lands slightly above ln(V)"
You will see about 5.34 below, against ln(V) of 4.83. Uniform guessing is
the best a model with no information can do, and a freshly initialized model
is not quite uniform — its random logits are confidently wrong about some
tokens, which costs more than spreading the probability evenly. So ln(V) is
a floor for ignorance, not an exact prediction. What identifies the failure
is that the number does not move.
def check_learning(run, label, min_drop=0.05):
drop = run.train_losses[0] - run.train_losses[-1]
return report(
f"loss is moving ({label})",
drop >= min_drop,
f"first={run.train_losses[0]:.3f} last={run.train_losses[-1]:.3f} "
f"drop={drop:.4f} (need >={min_drop}) guessing = {UNIFORM_LOSS:.3f}",
)
stuck = new_model()
stuck_run = train_model(
stuck,
loader(train_df),
loader(val_df, shuffle=False),
num_epochs=3,
config=CFG,
optimizer=optim.Adam(stuck.parameters(), lr=0.0), # <-- the bug
)
print()
check_learning(stuck_run, "lr=0.0")
check_learning(healthy_run, "the healthy run")
Localizing it: which parameters can actually learn?¶
"Loss is flat" tells you something is wrong, not what. One backward pass answers that, and it sorts every parameter into three buckets:
| Bucket | Meaning | Usual cause |
|---|---|---|
| frozen | requires_grad=False |
someone froze a submodule and forgot |
| dead | gradient is None or all zeros |
the graph was detached, or the output is unused |
| live | gradient is nonzero | healthy |
This needs no training at all — it runs on a freshly built model in under a second. Below we freeze the encoder and watch it get named.
def gradient_report(model, batch):
"""Sort parameters into frozen / dead / live after one backward pass."""
source, target = batch
criterion = nn.CrossEntropyLoss(ignore_index=vocab.pad_idx)
model.zero_grad(set_to_none=True)
logits = model(source, target[:, :-1])
criterion(logits.reshape(-1, logits.size(-1)), target[:, 1:].reshape(-1)).backward()
frozen, dead, live = [], [], []
for name, p in model.named_parameters():
if not p.requires_grad:
frozen.append(name)
elif p.grad is None or float(p.grad.abs().max()) == 0.0:
dead.append(name)
else:
live.append(name)
return frozen, dead, live
def check_gradients(model, batch, label):
frozen, dead, live = gradient_report(model, batch)
detail = f"live={len(live)} frozen={len(frozen)} dead={len(dead)}"
if frozen:
detail += f"\n first frozen: {frozen[0]}"
if dead:
detail += f"\n first dead: {dead[0]}"
return report(f"gradients reach every parameter ({label})", not frozen and not dead, detail)
batch = next(iter(loader(train_df)))
check_gradients(new_model(), batch, "fresh model")
frozen_encoder = new_model()
for parameter in frozen_encoder.transformer.encoder.parameters():
parameter.requires_grad_(False) # <-- the bug
check_gradients(frozen_encoder, batch, "encoder frozen")
Recognize it: the loss is flat, and near ln(V).
Identify it with the two checks above, in this order:
check_gradientsreports frozen parameters → someone calledrequires_grad_(False), or built the optimizer over a filtered parameter list.- It reports dead parameters → the graph is detached somewhere, or that output never reaches the loss.
- Everything is live but the loss still doesn't move → gradients exist and
are being discarded. Check that
optimizer.step()is actually called, and that the learning rate is not zero — which is the bug in the cell above, and why it reports every parameter live.
!!! note "A frozen encoder is subtler than it sounds" It does not flatten the loss. A randomly initialized encoder is still a fixed random projection of the input, and the decoder can learn to read it — so training reaches roughly 0.74 validation loss here instead of the healthy 0.10. Translations come out mostly right with the numbers wrong, because precise token identity is what a random projection loses first. Partial damage looks like a merely disappointing model, which is why the gradient check earns its place: it is the difference between "needs more epochs" and "26 parameters were never going to move."
Question 3 — Is it learning the wrong thing?¶
Now the loss falls, beautifully, all the way down. On the training set.
The tell is the sign of the gap between training and validation loss. Look at the healthy run: validation came out below training loss. That is not a fluke — dropout is active during training and disabled during validation, so a healthy model usually scores slightly better on validation.
When that ordering flips and keeps going, the model is memorizing.
def check_generalization(run, label, max_gap=0.30):
gap = run.val_losses[-1] - run.train_losses[-1]
return report(
f"generalizes ({label})",
gap <= max_gap,
f"train={run.train_losses[-1]:.3f} val={run.val_losses[-1]:.3f} "
f"gap={gap:+.3f} (need <=+{max_gap})",
)
seen = train_df.head(60) # <-- the bug: far too little data
memorizer = new_model()
memorizer_run = train_model(
memorizer,
loader(seen, batch_size=16),
loader(val_df, shuffle=False),
num_epochs=150,
config=CFG,
optimizer=optim.Adam(memorizer.parameters(), lr=1e-3),
)
print()
check_generalization(healthy_run, "3000 pairs")
check_generalization(memorizer_run, "60 pairs, 150 epochs")
# The single number above is the end of a story. Watch it develop:
print("\n epoch train val gap")
for epoch in (10, 30, 60, 100, 150):
i = epoch - 1
train_i, val_i = memorizer_run.train_losses[i], memorizer_run.val_losses[i]
print(f" {epoch:5d} {train_i:.3f} {val_i:.3f} {val_i - train_i:+.3f}")
Recognize it: validation loss sits clearly above training loss, and the gap widens as training continues — in the table above it runs +0.03, +0.12, +0.66, +1.08. Watching only the training curve hides this completely: that curve looks better than the healthy model's, all the way down.
Note the first two rows. Early on the gap is near zero, and briefly negative — the memorizing run and the healthy one are indistinguishable by this check at epoch 10. Overfitting is not a state the model is in from the start; it is something it does to itself over time, which is why you watch the gap rather than test it once.
Fix it: more data first; then regularization (dropout, label smoothing) and early stopping on validation loss. Note that early stopping requires a validation set the model never trains on — which is the subject of the next question.
Question 4 — Is the measurement lying?¶
The most dangerous failure is the one that makes your numbers look good.
The model from Question 3 memorized 60 sentence pairs. Score it on those same pairs and it looks outstanding. Score it on fresh ones and it collapses. Both numbers come from the same model and the same metric.
def bleu_on(model, frame):
return compute_bleu(translate(model, list(frame.src)), list(frame.tgt)).score
fresh = make_corpus(60, seed=4321)
mixed = pd.concat([seen, fresh]).reset_index(drop=True)
print(f"BLEU on the 60 pairs it trained on : {bleu_on(memorizer, seen):5.2f}")
print(f"BLEU on 60 pairs it has never seen : {bleu_on(memorizer, fresh):5.2f}")
print(f"BLEU on a 50/50 mix of the two : {bleu_on(memorizer, mixed):5.2f} <- true of neither")
A contaminated test set does not report an error. It reports a number, halfway between the truth and a lie, and you have no way to tell from the number alone.
The diagnostic is to compare the sets directly, before trusting any score:
def check_contamination(test_frame, train_frame, label):
shared = set(test_frame.src) & set(train_frame.src)
return report(
f"test set is clean ({label})",
not shared,
f"{len(shared)}/{len(test_frame)} test sources also appear in training",
)
check_contamination(mixed, seen, "the 50/50 mix")
check_contamination(fresh, seen, "the fresh pairs")
Recognize it: a score that is much better than the model's behaviour when you read its actual output, or a suspicious jump after a data change.
Fix it: split before you do anything else, deduplicate across the split, and re-run the overlap check whenever the corpus changes. Deduplicate on the source side at minimum — near-duplicates that differ only in whitespace or casing will slip past the exact-match check above.
!!! warning "This generalizes beyond contamination" Any comparison where more than one thing changed produces a number that is true of nothing. This project shipped a claim that a larger corpus was worth +2.33 BLEU, when the new model had also trained 80% longer; the corpus was worth about +0.29, with an error bar crossing zero. Change one variable at a time, or you are measuring their sum.
Question 5 — Is it the environment?¶
Everything above is about the model. This one is about the seven lines around it.
The canonical version: inference with the model still in training mode. Dropout stays active, so it randomly zeroes activations and rescales the survivors. The model still runs. It still produces plausible output. Every number it gives you is wrong, and a different kind of wrong each time you ask.
criterion = nn.CrossEntropyLoss(ignore_index=vocab.pad_idx)
def validation_loss(model):
total = batches = 0
for source, target in loader(val_df, shuffle=False):
with torch.no_grad():
logits = model(source, target[:, :-1])
total += criterion(
logits.reshape(-1, logits.size(-1)), target[:, 1:].reshape(-1)
).item()
batches += 1
return total / batches
healthy.train()
in_train_mode = [round(validation_loss(healthy), 4) for _ in range(3)]
healthy.eval()
in_eval_mode = [round(validation_loss(healthy), 4) for _ in range(3)]
print(f"same model, same data, train() mode: {in_train_mode}")
print(f"same model, same data, eval() mode: {in_eval_mode}")
def check_eval_mode(model, label):
return report(
f"model is in eval mode ({label})",
not model.training,
f"model.training={model.training} "
f"({'dropout is active — results will not reproduce' if model.training else 'deterministic'})",
)
print()
healthy.train()
check_eval_mode(healthy, "before eval()")
healthy.eval()
check_eval_mode(healthy, "after eval()")
Recognize it: two identical runs disagree. That is the signature, and it is worth more than any single number — a measurement you cannot repeat is not a measurement. Note that the train-mode loss here is not merely noisy, it is roughly 2.5x the true value, so it is also badly biased.
Fix it: call model.eval() before any evaluation, decoding, or attention
plot, and wrap inference in torch.no_grad(). If you need a stronger guarantee,
assert it — check_eval_mode above is two lines.
The same class of bug covers a stale checkpoint, a model on a different device than its inputs, and a vocabulary rebuilt after the checkpoint was saved. Each one leaves the code running and the results meaningless.
The procedure¶
| # | Question | Symptom | Check | Fix |
|---|---|---|---|---|
| 1 | Is the data right? | loss falls but plateaus high; fluent output unrelated to input | diagnose_alignment(frame).looks_aligned() |
realign or rebuild the corpus |
| 2 | Is it learning at all? | loss flat near ln(V) |
check_learning, then check_gradients |
unfreeze / reattach / fix the optimizer |
| 3 | Is it learning the wrong thing? | val loss above train loss, gap widening | check_generalization |
more data, regularization, early stopping |
| 4 | Is the measurement lying? | scores too good for the output you read | check_contamination |
split and deduplicate first |
| 5 | Is it the environment? | two identical runs disagree | check_eval_mode |
model.eval() and torch.no_grad() |
Three habits are worth more than the table:
- Know your reference points.
ln(V)is what guessing scores. A healthy validation loss sits at or slightly below training loss. Without those you cannot tell a bad number from a fine one. - Keep a control. Nearly every check above is a comparison. The single most useful artifact when something breaks is a run you know was healthy.
- Change one thing at a time. Two changes and one number measures their sum, which is true of neither.
Try it yourself¶
The checks are the point, so use them on a break you haven't seen:
- Swap
srcandtgtfor half the corpus. Which check catches it? (Note that length correlation will not — both sides stay the same length.) - Build the vocabulary from
train_dfonly, then evaluate onval_df, whose numbers include types the vocabulary has never seen. - Set
num_epochs=200on the full corpus and watch for wherecheck_generalizationstarts to fail. - Replace
argmaxdecoding with sampling and re-run Question 5's repeatability check. What does "two runs disagree" mean when the decoder is stochastic on purpose?
For the six failures this project actually shipped, and what each one cost, read When it fails.