Tutorial 5: Translating Sentences It Has Never Seen¶
Every tutorial so far trained on a toy corpus — twelve phrases, or a synthetic reversal task — and then translated sentences from that same corpus. The models worked, and none of them had ever produced a translation of something they had not memorized.
This one does. It loads a model trained on 64,311 real English-Spanish sentence pairs and points it at talks it has never seen.
A warning before you start: the translations are not good. That is the honest result of this much data and a few minutes of training, and seeing exactly how it is not good is the point of this tutorial. A model that memorized twelve phrases looks perfect and teaches you nothing about translation.
# Install TorchLingo (uncomment in Google Colab)
# %pip install torchlingo
from pathlib import Path
import pandas as pd
import torch
from torchlingo.data_processing.vocab import SentencePieceVocab
from torchlingo.inference import beam_search_decode, greedy_decode
from torchlingo.models import SimpleTransformer
PRETRAINED = Path("data/pretrained")
device = torch.device("cpu")
checkpoint = torch.load(PRETRAINED / "model.pt", map_location=device, weights_only=False)
vocab = SentencePieceVocab(str(PRETRAINED / "spm.model"))
model = SimpleTransformer(
src_vocab_size=checkpoint["vocab_size"],
tgt_vocab_size=checkpoint["vocab_size"],
**checkpoint["model_config"],
).to(device)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
print(f"trained on {checkpoint['train_pairs']:,} sentence pairs "
f"for {checkpoint['trained_minutes']} minutes")
print(f"final validation loss: {checkpoint['val_losses'][-1]:.3f}")
print(f"vocabulary: {len(vocab):,} subword pieces")
trained on 54,072 sentence pairs for 11.1 minutes final validation loss: 4.035 vocabulary: 3,000 subword pieces
The held-out set is genuinely held out¶
data/pretrained/test.tsv comes from talks the model never saw — not random
sentences pulled out of talks it trained on.
That distinction does most of the work here. Consecutive sentences in a transcript share a speaker, a topic and a vocabulary, so holding out random sentences leaves the model with the surrounding context of every one of them. Scores taken that way look much better than the model deserves.
Holding out whole talks means every sentence below is about a subject, in a voice, the model has never encountered.
test = pd.read_csv(PRETRAINED / "test.tsv", sep="\t", dtype=str, keep_default_na=False)
print(f"{len(test):,} held-out sentence pairs\n")
def translate(sentence, beam_size=None):
"""Translate one sentence, greedy by default."""
ids = vocab.encode(sentence, add_special_tokens=True)
src = torch.tensor([ids]).to(device)
if beam_size is None:
tokens = greedy_decode(model, src, max_len=60)[0]
else:
tokens = beam_search_decode(model, src, beam_size=beam_size, max_len=60)
return vocab.decode(tokens, skip_special_tokens=True)
sample = test.sample(6, random_state=0)
for _, row in sample.iterrows():
print(f"EN {row['src']}")
print(f"MODEL {translate(row['src'])}")
print(f"REF {row['tgt']}")
print()
1,889 held-out sentence pairs EN It's more like, say, if you were a dog. MODEL Es como si se le preguntan, si se le preguntan. REF Es como, digamos, si fueras un perro. EN This is where I come from, Todmorden. MODEL Este es donde estoy de dónde. REF Yo vengo de Todmorden. EN Because commercial companies are inherently designed to get as much out of you [as] they can get away with.
MODEL Porque se pueden usar mucho más grandes problemas de diseño de diseño que pueden hacer con los compañías. REF Porque las empresas están pensadas en esencia para conseguir lo máximo que puedan de nosotros. EN So if all human life depends on plants, doesn't it make sense that perhaps we should try to save them?
MODEL Entonces, ¿cómo deberíamos hacer eso? ¿cómo deberíamos hacer eso? REF Así que si toda la vida humana depende de las plantas, ¿acaso no tendría sentido que intentáramos salvarlas? EN BG: Somebody in the audience asked me, How does he breathe up there? MODEL BG: Algunas me preguntó: ¿Cómo me preguntó en el Brio? REF BG: Alguien de la audiencia me preguntó ¿Cómo hace para respirar allá arriba?, EN So the biggest and the most important thing is our neutral point-of-view policy. MODEL Así que la mayoría de la mayoría de la mayoría de la mayoría de la mayoría de la mayoría de los adultos y la mayoría de la mayoría. REF Lo más importante es nuestra política del punto de vista neutral.
Read those carefully¶
This is the part worth slowing down for.
It is producing Spanish. Word order, agreement, function words, inverted
question marks. Nothing told it what Spanish is; it learned that from
examples. Look at the fifth pair: it even kept the BG: speaker tag, because
TED transcripts are full of them.
And then look at what goes wrong. The dominant failure is not invention, it is repetition:
la mayoría de la mayoría de la mayoría de la mayoría...
¿cómo deberíamos hacer eso? ¿cómo deberíamos hacer eso?
Es como si se le preguntan, si se le preguntan.
This is the classic degenerate mode of an undertrained sequence model, and it has a mechanical explanation. At every step the decoder picks the most probable next token given what it has already produced. When the model is unsure of the content, the safest continuation is often the phrase it just emitted — that phrase is, after all, demonstrably likely in this context. Greedy decoding has no memory of having said it before and no way to prefer novelty.
Two things in this library exist partly because of that failure: beam search,
which can prefer a different path, and the length penalty alpha, which
stops the search from being rewarded for producing more of the same. You will
see beam search earn its keep further down.
Fluency and correctness are separate properties. This model is small enough that you can watch them come apart.
Putting a number on it¶
BLEU compares the model's output to the reference translation by n-gram overlap. Tutorial 3 warned you not to trust its BLEU of 100, because the model had memorized its test set. This one is measured on unseen talks, so the number means something.
It will be low — around 7. For reference, the scale in Tutorial 3 calls anything under 10 "almost unusable", and production systems on English-Spanish score in the 40s.
A number that low is not a failure of the exercise. It is the measurement working: the translations you just read are almost unusable, and BLEU says so. Tutorial 3's score of 100 was the broken measurement, not this one.
from sacrebleu.metrics import BLEU
subset = test.sample(200, random_state=1)
hypotheses = [translate(s) for s in subset["src"]]
references = [list(subset["tgt"])]
bleu = BLEU()
score = bleu.corpus_score(hypotheses, references)
print(score)
BLEU = 4.39 33.9/7.9/2.1/0.6 (BP = 1.000 ratio = 1.068 hyp_len = 2918 ref_len = 2732)
Greedy versus beam search, on a model that is actually uncertain¶
Tutorial 3 compared the two and found no difference on any sentence, because a model that has memorized its answers has nothing to be uncertain about. Every beam followed the same path.
This model is uncertain about nearly everything, so the comparison finally has something to show.
compare = test.sample(60, random_state=2)
greedy_out = [translate(s) for s in compare["src"]]
beam_out = [translate(s, beam_size=5) for s in compare["src"]]
refs = [list(compare["tgt"])]
differ = sum(g != b for g, b in zip(greedy_out, beam_out))
print(f"greedy BLEU {bleu.corpus_score(greedy_out, refs).score:.2f}")
print(f"beam-5 BLEU {bleu.corpus_score(beam_out, refs).score:.2f}")
print(f"different translations: {differ}/{len(compare)}")
print()
for src, g, b in list(zip(compare["src"], greedy_out, beam_out))[:5]:
if g == b:
continue
print(f"EN {src[:70]}")
print(f"greedy {g[:70]}")
print(f"beam-5 {b[:70]}")
print()
greedy BLEU 2.71 beam-5 BLEU 2.77 different translations: 55/60 EN You could change the genes in principle. greedy Podrían cambiar el cambio en el cambio climático. beam-5 Podrían cambiar el cambio climático. EN It hooks up all parts of the brain. greedy Es parte de las cerebro. beam-5 Es parte del cerebro. EN So the best place for x-ray crystallography was at the Cavendish Labor greedy Así que el lugar de la mejoración de Cratratratratratratal Cratratratr beam-5 Así que la mejor de Cratratalla de Cratratalla de Cratalla de Cratrata EN Across age, across income, across culture. greedy Un ciento, una cultura, una cultura, una cultura. beam-5 Una cultura, una cultura, una cultura, una estructura. EN How beautiful it is to lose ourselves in these little streets on the i greedy Es la razón en nuestra página en nuestra corazón. beam-5 Espero que es en nuestras vidas en nuestras vidas.
First, why is that BLEU lower than the one above? Different sample. The earlier number used 200 sentences; this comparison uses 60, to keep beam search from dominating the runtime. BLEU on 60 short sentences is noisy, and the absolute value moves around by a point or more depending on which ones you draw.
That is worth internalising: a BLEU score without a stated test set and size is not a number you can compare against anything. What is meaningful here is the difference between the two rows, because both were measured on identical inputs.
Even so, do not read too much into the gap on 60 sentences. Measured properly — 200 sentences, five different samples, differences taken within each sample — beam search is worth about +1.6 BLEU over greedy on this model, and the error bars are in Decoding. Sixty sentences is enough to see the behaviour and not enough to size it.
Beam search is not magic. It is a real but modest improvement that costs several times the computation, and whether it is worth paying depends on what you are doing with the output.
Look at what it fixes, though. In runs of this model it tends to repair exactly the failure described above:
greedy Podrían cambiar el cambio en el cambio climático.
beam-5 Podrían cambiar el cambio climático.
Greedy walked into a repetition and could not get out. Beam search was carrying an alternative hypothesis that had not, and that one finished with a better normalized score. This is the mechanism from Tutorial 3 doing visible work for the first time.
Your turn: train one yourself¶
Loading a checkpoint shows you the destination. Training shows you the road.
The cell below trains a model from scratch for a few hundred steps on a slice of the same data. It will be much worse than the one above — that one had twenty epochs over the whole corpus, this gets a fraction of one.
The point is to see what undertrained looks like, so you can recognise it. An undertrained translation model does not produce nonsense; it produces the most common output it can, over and over.
from functools import partial
from torch import optim
from torch.utils.data import DataLoader
from torchlingo.config import Config
from torchlingo.data_processing.batching import collate_fn
from torchlingo.data_processing.dataset import NMTDataset
corpus = pd.read_csv("data/example.tsv", sep="\t", dtype=str, keep_default_na=False)
corpus = corpus[corpus["kind"] == "transcript"].head(4000)
corpus[["src", "tgt"]].to_csv("data/quick_train.tsv", sep="\t", index=False)
cfg = Config(batch_size=32)
dataset = NMTDataset(Path("data/quick_train.tsv"), src_vocab=vocab, tgt_vocab=vocab,
max_length=60)
loader = DataLoader(dataset, batch_size=32, shuffle=True,
collate_fn=partial(collate_fn, pad_idx=vocab.pad_idx))
torch.manual_seed(0)
quick = SimpleTransformer(
src_vocab_size=len(vocab), tgt_vocab_size=len(vocab),
**checkpoint["model_config"],
).to(device)
from torchlingo.training import train_model
train_model(quick, loader, num_epochs=1, config=cfg,
optimizer=optim.Adam(quick.parameters(), lr=3e-4))
quick.eval()
for sentence in list(test["src"].head(3)):
ids = vocab.encode(sentence, add_special_tokens=True)
out = greedy_decode(quick, torch.tensor([ids]).to(device), max_len=40)[0]
print(f"EN {sentence[:66]}")
print(f"UNDERTRAINED {vocab.decode(out, skip_special_tokens=True)[:66]}")
print()
Epoch 1 Step 100/125 | Train Loss: 8.0284
Epoch 1/1 | Train: 8.0106 EN I want to help you re-perceive what philanthropy is, what it could UNDERTRAINED bre beautifbre beautifbre beautifbre beautifbre beautifbre beautif EN I want to start with these word pairs here. UNDERTRAINED brebrebrebrebrebrebrebrebrebrebrebrebrebrebrebrebrebrebrebrebrebre
EN We all know which side of these we'd like to be on. UNDERTRAINED breiendoiouslyiouslyiouslyiouslyiouslyiouslyiouslyiouslyiouslyious
Compare that to the pretrained model's output earlier. The undertrained one has learned the shape of the output — Spanish-looking tokens, plausible length — without learning what any particular sentence means. It has found the safest guess and is repeating it.
That is what the first few hundred steps of every translation model look like.
What would make this better¶
In rough order of how much they would buy:
| More training | Thirty-six epochs in half an hour. Real systems train for days. |
| More data | 64k pairs is small. Production systems use tens of millions. |
| A bigger model | 2.5M parameters. Production models are 100-1000× larger. |
| Better data | TED transcripts are one narrow domain of spoken register. |
Notice that model size is third. It is the lever people reach for first and it is rarely the binding constraint at this scale.
How we know that order, and how we got it wrong first¶
That table used to lead with more data, and the reason is a mistake worth walking through, because it is the most common way a measurement lies to you.
This checkpoint was retrained after the corpus grew from 53,520 to 64,311 pairs, when a sentence aligner recovered 98 talks that had been discarded. The new model scored +2.33 BLEU on a held-out set pinned to be identical. Paired bootstrap, 300 resamples, winning 100% of them. That looks conclusive.
It was wrong. The new model had also trained for 36 epochs against the old one's 20, because of a bug that made a checkpoint's recorded epoch count look like 36 when it was 20. Two things changed, and the writeup credited one.
Running the missing arm of the experiment — the old corpus at 36 epochs — separates them:
| data | epochs | BLEU |
|---|---|---|
| 53,520 pairs | 20 | 4.96 |
| 53,520 pairs | 36 | 7.01 |
| 64,311 pairs | 36 | 7.32 |
epochs 20 -> 36, data held fixed: +2.05 BLEU
+20% data, epochs held fixed: +0.29 ± 0.22 95% CI [-0.16, +0.71]
The data effect's confidence interval crosses zero. Nearly all of the original +2.33 was training length, and what remains cannot be distinguished from noise. The model was not data-starved. It was undertrained, and 13,348 extra sentence pairs bought nothing we can measure.
Three things worth taking from that.
Error bars do not make a measurement controlled. The original had five seeds, paired comparisons, a bootstrap and a pinned test set, and it was still measuring the wrong thing. Those techniques tell you whether a difference is real. They cannot tell you what caused it — only the experiment's design can, and here the design had two variables moving at once.
The boring hypothesis deserved a turn first. "We stopped training too early" is less interesting than "the model needs more data", which is probably why it was not checked. Undertraining is also the single most common reason a small model underperforms, so it should have been the first suspect, not the one found by accident.
Ask what the comparison holds fixed. Not "is this difference significant" but "if I am wrong about the cause, what would that look like?" Here it would look exactly like what was observed, which is the tell.
The numbers above live in docs/docs/_generated/checkpoint_comparison.json,
including both the controlled and the confounded run, and
scripts/compare_checkpoints.py reproduces them.
Summary¶
You have now seen a model translate text it was never trained on, and seen exactly how far short of useful it falls. Both halves matter: the mechanics work end to end, and the gap between "the mechanics work" and "this is a translation system" is enormous.
Every previous tutorial demonstrated the first half only.
What's Next?¶
- Tutorial 4: Attention and Alignment — how attention works, on an LSTM
- Read Decoding for what beam search and
alphaactually buy, measured on this model - Read When It Fails for diagnosing a model that is not working
- Read Vocabulary for why this uses subwords, measured
- Retrain the checkpoint yourself with
python scripts/train_example_model.py