Tutorial 3: Inference and Beam Search¶

Generate translations using greedy and beam search decoding strategies.

⚠️ Prerequisites: This tutorial requires the model checkpoint from Tutorial 2. Run Tutorial 2 first!

⚡ Running in Google Colab? Make sure to:

1. Go to Runtime → Change runtime type → GPU (optional)
2. Uncomment and run the %pip install torchlingo cell below
3. Run Tutorial 2 first to create the model checkpoint

In [1]:

# Install TorchLingo (uncomment in Google Colab)
# %pip install torchlingo

# Check GPU availability
import torch
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")

# Install TorchLingo (uncomment in Google Colab) # %pip install torchlingo # Check GPU availability import torch print(f"PyTorch version: {torch.__version__}") print(f"CUDA available: {torch.cuda.is_available()}") if torch.cuda.is_available(): print(f"GPU: {torch.cuda.get_device_name(0)}")

PyTorch version: 2.13.0
CUDA available: False

In [2]:

# Set device (will use GPU if available)
device = torch.device("cuda" if torch.cuda.is_available() else 
                      "mps" if torch.backends.mps.is_available() else "cpu")
print(f"Using device: {device}")

# Import TorchLingo modules
from pathlib import Path
import torch.nn.functional as F
from torchlingo.models import SimpleTransformer

print("✓ Imports successful!")

# Set device (will use GPU if available) device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu") print(f"Using device: {device}") # Import TorchLingo modules from pathlib import Path import torch.nn.functional as F from torchlingo.models import SimpleTransformer print("✓ Imports successful!")

Using device: mps

✓ Imports successful!

In [3]:

# Load the model from Tutorial 2
ckpt_path = Path("checkpoints/tiny_model.pt")

# Stop here rather than carrying on: without the checkpoint, `model` and the
# vocabs are never defined, and the notebook would otherwise fail several cells
# later with a confusing `NameError` instead of this explanation.
if not ckpt_path.exists():
    raise FileNotFoundError(
        f"No checkpoint found at {ckpt_path}. Run Tutorial 2 first — it trains "
        "the model this tutorial decodes with and saves it to that path."
    )

# weights_only=False because the checkpoint stores pickled vocab objects
# alongside the weights (PyTorch >= 2.6 refuses these by default).
# Only do this for files you created yourself or fully trust.
checkpoint = torch.load(ckpt_path, map_location=device, weights_only=False)

model = SimpleTransformer(
    src_vocab_size=len(checkpoint['src_vocab']),
    tgt_vocab_size=len(checkpoint['tgt_vocab']),
    **checkpoint['config'],
).to(device)

model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

src_vocab = checkpoint['src_vocab']
tgt_vocab = checkpoint['tgt_vocab']

print("✓ Model loaded!")

# Load the model from Tutorial 2 ckpt_path = Path("checkpoints/tiny_model.pt") # Stop here rather than carrying on: without the checkpoint, `model` and the # vocabs are never defined, and the notebook would otherwise fail several cells # later with a confusing `NameError` instead of this explanation. if not ckpt_path.exists(): raise FileNotFoundError( f"No checkpoint found at {ckpt_path}. Run Tutorial 2 first — it trains " "the model this tutorial decodes with and saves it to that path." ) # weights_only=False because the checkpoint stores pickled vocab objects # alongside the weights (PyTorch >= 2.6 refuses these by default). # Only do this for files you created yourself or fully trust. checkpoint = torch.load(ckpt_path, map_location=device, weights_only=False) model = SimpleTransformer( src_vocab_size=len(checkpoint['src_vocab']), tgt_vocab_size=len(checkpoint['tgt_vocab']), **checkpoint['config'], ).to(device) model.load_state_dict(checkpoint['model_state_dict']) model.eval() src_vocab = checkpoint['src_vocab'] tgt_vocab = checkpoint['tgt_vocab'] print("✓ Model loaded!")

✓ Model loaded!

Part 1: Greedy Decoding (Review)¶

Greedy decoding picks the most likely token at each step.

In [4]:

def greedy_decode(model, src_sentence, src_vocab, tgt_vocab, device, max_len=20):
    """Generate translation using greedy decoding.
    
    At each step, pick the single most likely next token.
    """
    model.eval()
    
    # Encode source
    src_indices = src_vocab.encode(src_sentence, add_special_tokens=True)
    src_tensor = torch.tensor([src_indices]).to(device)
    
    with torch.no_grad():
        memory = model.encode(src_tensor)
    
    # Decode
    output_indices = [tgt_vocab.sos_idx]
    
    for _ in range(max_len):
        tgt_tensor = torch.tensor([output_indices]).to(device)
        
        with torch.no_grad():
            logits = model.decode(tgt_tensor, memory)
        
        # Greedy: pick argmax
        next_token = logits[0, -1, :].argmax().item()
        output_indices.append(next_token)
        
        if next_token == tgt_vocab.eos_idx:
            break
    
    return tgt_vocab.decode(output_indices, skip_special_tokens=True)

def greedy_decode(model, src_sentence, src_vocab, tgt_vocab, device, max_len=20): """Generate translation using greedy decoding. At each step, pick the single most likely next token. """ model.eval() # Encode source src_indices = src_vocab.encode(src_sentence, add_special_tokens=True) src_tensor = torch.tensor([src_indices]).to(device) with torch.no_grad(): memory = model.encode(src_tensor) # Decode output_indices = [tgt_vocab.sos_idx] for _ in range(max_len): tgt_tensor = torch.tensor([output_indices]).to(device) with torch.no_grad(): logits = model.decode(tgt_tensor, memory) # Greedy: pick argmax next_token = logits[0, -1, :].argmax().item() output_indices.append(next_token) if next_token == tgt_vocab.eos_idx: break return tgt_vocab.decode(output_indices, skip_special_tokens=True)

In [5]:

# Test greedy decoding
test_sentence = "Hello world"
translation = greedy_decode(model, test_sentence, src_vocab, tgt_vocab, device)
print(f"Greedy: '{test_sentence}' → '{translation}'")

# Test greedy decoding test_sentence = "Hello world" translation = greedy_decode(model, test_sentence, src_vocab, tgt_vocab, device) print(f"Greedy: '{test_sentence}' → '{translation}'")

Greedy: 'Hello world' → 'Hola mundo'

Greedy Limitations¶

Greedy decoding can get stuck in suboptimal paths:

Step 1: P("El") = 0.4, P("La") = 0.35, P("Un") = 0.25
        → Pick "El" (highest)
        
Step 2: P("gato"|"El") = 0.3, P("perro"|"El") = 0.25, ...
        → But maybe "La casa" would have been better overall!

Greedy only considers one path—it can't backtrack.

Part 2: Beam Search¶

Beam search keeps track of multiple hypotheses ("beams") and picks the best complete sequence.

In [6]:

def beam_search_decode(
    model, src_sentence, src_vocab, tgt_vocab, device,
    beam_size=3, max_len=20, length_penalty=0.6
):
    """Generate translation using beam search.
    
    Keeps beam_size hypotheses at each step and returns the best one.
    
    Args:
        beam_size: Number of hypotheses to keep
        length_penalty: Penalize/reward longer sequences (alpha in paper)
    """
    model.eval()
    
    # Encode source
    src_indices = src_vocab.encode(src_sentence, add_special_tokens=True)
    src_tensor = torch.tensor([src_indices]).to(device)
    
    with torch.no_grad():
        memory = model.encode(src_tensor)
    
    # Initialize beams: (sequence, log_prob)
    beams = [([tgt_vocab.sos_idx], 0.0)]
    completed = []
    
    for _ in range(max_len):
        all_candidates = []
        
        for seq, score in beams:
            # Skip completed sequences
            if seq[-1] == tgt_vocab.eos_idx:
                completed.append((seq, score))
                continue
            
            # Get probabilities for next token
            tgt_tensor = torch.tensor([seq]).to(device)
            
            with torch.no_grad():
                logits = model.decode(tgt_tensor, memory)
            
            log_probs = F.log_softmax(logits[0, -1, :], dim=-1)
            
            # Get top beam_size candidates
            topk_log_probs, topk_indices = log_probs.topk(beam_size)
            
            for log_prob, idx in zip(topk_log_probs, topk_indices):
                new_seq = seq + [idx.item()]
                new_score = score + log_prob.item()
                all_candidates.append((new_seq, new_score))
        
        # Keep top beam_size candidates
        all_candidates.sort(key=lambda x: x[1], reverse=True)
        beams = all_candidates[:beam_size]
        
        # Stop if all beams are completed
        if not beams:
            break
    
    # Add any remaining beams to completed
    completed.extend(beams)
    
    # Apply length penalty and pick best
    def score_with_length_penalty(seq, score):
        length = len(seq)
        return score / (length ** length_penalty)
    
    best_seq, best_score = max(
        completed, 
        key=lambda x: score_with_length_penalty(x[0], x[1])
    )
    
    return tgt_vocab.decode(best_seq, skip_special_tokens=True)

def beam_search_decode( model, src_sentence, src_vocab, tgt_vocab, device, beam_size=3, max_len=20, length_penalty=0.6 ): """Generate translation using beam search. Keeps beam_size hypotheses at each step and returns the best one. Args: beam_size: Number of hypotheses to keep length_penalty: Penalize/reward longer sequences (alpha in paper) """ model.eval() # Encode source src_indices = src_vocab.encode(src_sentence, add_special_tokens=True) src_tensor = torch.tensor([src_indices]).to(device) with torch.no_grad(): memory = model.encode(src_tensor) # Initialize beams: (sequence, log_prob) beams = [([tgt_vocab.sos_idx], 0.0)] completed = [] for _ in range(max_len): all_candidates = [] for seq, score in beams: # Skip completed sequences if seq[-1] == tgt_vocab.eos_idx: completed.append((seq, score)) continue # Get probabilities for next token tgt_tensor = torch.tensor([seq]).to(device) with torch.no_grad(): logits = model.decode(tgt_tensor, memory) log_probs = F.log_softmax(logits[0, -1, :], dim=-1) # Get top beam_size candidates topk_log_probs, topk_indices = log_probs.topk(beam_size) for log_prob, idx in zip(topk_log_probs, topk_indices): new_seq = seq + [idx.item()] new_score = score + log_prob.item() all_candidates.append((new_seq, new_score)) # Keep top beam_size candidates all_candidates.sort(key=lambda x: x[1], reverse=True) beams = all_candidates[:beam_size] # Stop if all beams are completed if not beams: break # Add any remaining beams to completed completed.extend(beams) # Apply length penalty and pick best def score_with_length_penalty(seq, score): length = len(seq) return score / (length ** length_penalty) best_seq, best_score = max( completed, key=lambda x: score_with_length_penalty(x[0], x[1]) ) return tgt_vocab.decode(best_seq, skip_special_tokens=True)

In [7]:

# Test beam search
test_sentence = "Hello world"

greedy_result = greedy_decode(model, test_sentence, src_vocab, tgt_vocab, device)
beam_result = beam_search_decode(model, test_sentence, src_vocab, tgt_vocab, device, beam_size=3)

print(f"Input:  '{test_sentence}'")
print(f"Greedy: '{greedy_result}'")
print(f"Beam-3: '{beam_result}'")

# Test beam search test_sentence = "Hello world" greedy_result = greedy_decode(model, test_sentence, src_vocab, tgt_vocab, device) beam_result = beam_search_decode(model, test_sentence, src_vocab, tgt_vocab, device, beam_size=3) print(f"Input: '{test_sentence}'") print(f"Greedy: '{greedy_result}'") print(f"Beam-3: '{beam_result}'")

Input:  'Hello world'
Greedy: 'Hola mundo'
Beam-3: 'Hola mundo'

Part 3: Comparing Strategies¶

In [8]:

# Compare on multiple sentences
test_sentences = [
    "Hello world",
    "Good morning",
    "Thank you",
    "I love you",
    "The cat sleeps",
]

print(f"{'Input':<20} {'Greedy':<20} {'Beam-3':<20}")
print("-" * 60)

for src in test_sentences:
    greedy = greedy_decode(model, src, src_vocab, tgt_vocab, device)
    beam = beam_search_decode(model, src, src_vocab, tgt_vocab, device)
    print(f"{src:<20} {greedy:<20} {beam:<20}")

# Compare on multiple sentences test_sentences = [ "Hello world", "Good morning", "Thank you", "I love you", "The cat sleeps", ] print(f"{'Input':<20} {'Greedy':<20} {'Beam-3':<20}") print("-" * 60) for src in test_sentences: greedy = greedy_decode(model, src, src_vocab, tgt_vocab, device) beam = beam_search_decode(model, src, src_vocab, tgt_vocab, device) print(f"{src:<20} {greedy:<20} {beam:<20}")

Input                Greedy               Beam-3              
------------------------------------------------------------
Hello world          Hola mundo           Hola mundo          
Good morning         Buenos días          Buenos días         

Thank you            Gracias              Gracias             
I love you           Te amo               Te amo              

The cat sleeps       El gato duerme       El gato duerme      

In [ ]:

# Effect of beam size -- or rather, the absence of one.
test_sentence = "Hello world"

results = {}
for beam_size in [1, 2, 3, 5, 10]:
    results[beam_size] = beam_search_decode(
        model, test_sentence, src_vocab, tgt_vocab, device,
        beam_size=beam_size
    )

print(f"Input: '{test_sentence}'")
print("-" * 40)
for beam_size, result in results.items():
    print(f"Beam-{beam_size:2d}: '{result}'")

# The point of this cell is that every row is the same, so the notebook asserts
# it rather than leaving you to notice. If this ever fires, the model has become
# uncertain enough for beam width to matter and the explanation below it is no
# longer the right one -- which is worth knowing loudly.
distinct = set(results.values())
assert len(distinct) == 1, (
    f"expected one translation across all beam sizes, got {len(distinct)}: "
    f"{distinct}. The model is no longer decisive, so the discussion below "
    "this cell needs rewriting."
)

print()
print(f"{len(results)} beam sizes, {len(distinct)} distinct translation.")
print("Widening the beam from 1 to 10 changed nothing at all.")

# Effect of beam size -- or rather, the absence of one. test_sentence = "Hello world" results = {} for beam_size in [1, 2, 3, 5, 10]: results[beam_size] = beam_search_decode( model, test_sentence, src_vocab, tgt_vocab, device, beam_size=beam_size ) print(f"Input: '{test_sentence}'") print("-" * 40) for beam_size, result in results.items(): print(f"Beam-{beam_size:2d}: '{result}'") # The point of this cell is that every row is the same, so the notebook asserts # it rather than leaving you to notice. If this ever fires, the model has become # uncertain enough for beam width to matter and the explanation below it is no # longer the right one -- which is worth knowing loudly. distinct = set(results.values()) assert len(distinct) == 1, ( f"expected one translation across all beam sizes, got {len(distinct)}: " f"{distinct}. The model is no longer decisive, so the discussion below " "this cell needs rewriting." ) print() print(f"{len(results)} beam sizes, {len(distinct)} distinct translation.") print("Widening the beam from 1 to 10 changed nothing at all.")

A measurement that tells you nothing¶

Five identical rows. It is tempting to read that as "beam size does not matter," and that would be the wrong lesson from a correct observation.

What it actually means is that this experiment cannot answer the question. Tutorial 2 trained on twelve phrases and we are decoding one of them. The model has memorized the answer, so it is certain, and a certain model gives beam search nothing to search. You would see the same five rows whether beam width mattered enormously or not at all.

This is worth more than the result it failed to produce. A sweep that returns the same value for every setting is telling you about your setup, not about the parameter:

• the model may be too certain, as here
• the parameter may not be reaching the code you think it is
• the effect may be real but smaller than the resolution of your measurement

Distinguishing those is most of experimental debugging. The first question is never "what does this tell me about beam size" but "could this experiment have detected an effect if there were one?"

Here it could not. To measure the real thing you need a model that is genuinely unsure — one trained on real data, translating a sentence it has never seen. That measurement is in Decoding, run on the pretrained model from Tutorial 5 across several held-out samples. In short: greedy to any beam width is the large gain, quality peaks around beam 3 to 5, and widening further makes it measurably worse.

!!! note "This happened for real, twice" The first time that measurement was run, on a weaker checkpoint, it reported that no beam width was distinguishable from any other. The model was retrained on more data, the error bars shrank, and the peak-and-decline shape appeared.

So the earlier conclusion was the same mistake as reading these five rows as "beam
size does not matter" — one level up, and with error bars and five random samples
that made it look rigorous. "No difference detectable" is a statement about your
measurement before it is a statement about the world.

Watching the search prune¶

Every table so far shows only the winner. Beam search spends its whole budget on hypotheses that lose, and the case for paying that cost is entirely about paths greedy would never reach — which the output never shows you.

The library's beam_search_decode will record them if you hand it a list:

trace = []
tokens = beam_search_decode(model, src, beam_size=3, trace=trace)

Tracing changes nothing about the result. It only writes down what happened.

In [10]:

from torchlingo.inference import beam_search_decode as library_beam_search
from torchlingo.visualization import format_beam_search

sentence = "The cat sleeps"
ids = src_vocab.encode(sentence, add_special_tokens=True)

trace = []
tokens = library_beam_search(
    model, torch.tensor([ids]).to(device), beam_size=3, max_len=8, trace=trace
)

print(f"{sentence!r} -> {tgt_vocab.decode(tokens, skip_special_tokens=True)!r}")
print()
print(format_beam_search(trace, itos=tgt_vocab.idx2token, winner=tokens, top=3, max_steps=4))
print()
print("  >  kept, and on the path that eventually won")
print("  +  kept into the next step")
print("  .  pruned here")

from torchlingo.inference import beam_search_decode as library_beam_search from torchlingo.visualization import format_beam_search sentence = "The cat sleeps" ids = src_vocab.encode(sentence, add_special_tokens=True) trace = [] tokens = library_beam_search( model, torch.tensor([ids]).to(device), beam_size=3, max_len=8, trace=trace ) print(f"{sentence!r} -> {tgt_vocab.decode(tokens, skip_special_tokens=True)!r}") print() print(format_beam_search(trace, itos=tgt_vocab.idx2token, winner=tokens, top=3, max_steps=4)) print() print(" > kept, and on the path that eventually won") print(" + kept into the next step") print(" . pruned here")

'The cat sleeps' -> 'El gato duerme'

step 0
  > -0.031  <sos> El
  + -5.416  <sos> gato
  + -5.480  <sos> estás
step 1
  > -0.050  <sos> El gato
  + -5.028  <sos> El Amo
  + -5.070  <sos> El Hasta
    ... 6 more considered
step 2
  > -0.075  <sos> El gato duerme
  + -4.337  <sos> El gato corre
  + -4.548  <sos> El gato gato
    ... 6 more considered
step 3
  > -0.115  <sos> El gato duerme <eos>
  + -3.805  <sos> El gato duerme Hasta
  + -3.840  <sos> El gato duerme El
    ... 6 more considered

  >  kept, and on the path that eventually won
  +  kept into the next step
  .  pruned here

The > is at the top of every step, and it is not close — roughly -0.03 against -5.4 for the runner-up. This model is certain, so beam search follows exactly the path greedy would have taken, and every one of those extra hypotheses was wasted work.

That is the honest result on this model, and it is worth sitting with rather than skipping past: beam search bought us nothing here. Tutorial 2 trained on twelve phrases and we are decoding one of them, so the model has memorized the answer. A memorized model has no uncertainty for beam search to exploit.

The interesting case is a > sitting below a +: the eventual winner ranked second or third at that step and survived only because the beam was wide enough to carry it. Greedy would have committed to the higher-scoring option and had no way back. That is the situation beam search exists for, and you need a model that is genuinely unsure — one trained on real data, translating a sentence it has never seen — to produce it.

So the honest summary of this section is: you now have the instrument, and this model is the wrong patient for it. Point it at a real model and look for the crossover.

Part 4: Does this match the library?¶

The two functions above are written to be read. TorchLingo ships its own versions in torchlingo.inference, and you should use those in real work:

from torchlingo.inference import beam_search_decode, greedy_decode

Whenever a tutorial reimplements something the library provides, the two can drift apart without anyone noticing — and then you learn one algorithm while the library runs another. So let's check, rather than assume.

Three places the version above is deliberately simpler:

	This notebook	torchlingo.inference
Pruning	keeps the top beam_size by raw cumulative score	ranks by the length-normalized score
Length penalty	score / length ** α	score / ((5 + length) / 6) ** α (Wu et al., 2016)
Ties	log_probs.topk(k)	_canonical_topk, a documented tie-breaking rule

The third is the one worth understanding. torch.topk does not promise any particular order among equally-scored elements, so a beam search built on it can return different output on different devices — for the same model and the same input. The library defines an explicit rule instead: prefer the higher score; among exactly equal scores prefer the lower token IDs, position by position. Ties are common early in training and with padded inputs, so this matters more than it sounds.

None of those differences changes the answer on this model, which is decisive enough never to hit a tie. The cell below proves that — and will fail loudly if it ever stops being true.

In [11]:

from torchlingo.inference import beam_search_decode as library_beam_search
from torchlingo.inference import greedy_decode as library_greedy_decode


def library_greedy(sentence):
    """Translate one sentence with the library's greedy decoder."""
    ids = src_vocab.encode(sentence, add_special_tokens=True)
    tokens = library_greedy_decode(model, torch.tensor([ids]).to(device), max_len=20)[0]
    return tgt_vocab.decode(tokens, skip_special_tokens=True)


def library_beam(sentence, beam_size=3):
    """Translate one sentence with the library's beam search."""
    ids = src_vocab.encode(sentence, add_special_tokens=True)
    tokens = library_beam_search(
        model, torch.tensor([ids]).to(device), beam_size=beam_size, max_len=20
    )
    return tgt_vocab.decode(tokens, skip_special_tokens=True)


comparisons = 0
for sentence in test_sentences:
    ours = greedy_decode(model, sentence, src_vocab, tgt_vocab, device)
    theirs = library_greedy(sentence)
    assert ours == theirs, f"greedy disagrees on {sentence!r}: {ours!r} vs {theirs!r}"
    comparisons += 1

    for beam_size in (1, 3, 5):
        ours = beam_search_decode(
            model, sentence, src_vocab, tgt_vocab, device, beam_size=beam_size
        )
        theirs = library_beam(sentence, beam_size)
        assert ours == theirs, (
            f"beam-{beam_size} disagrees on {sentence!r}: {ours!r} vs {theirs!r}"
        )
        comparisons += 1

print(f"{comparisons} comparisons, notebook and library agree on every one.")
print()
print("Note what this does and does not show. It shows the two implementations")
print("return the same translations for THESE sentences and THIS model. It does")
print("not show they are the same algorithm — they are not, as the table above")
print("says. Use the library's version when the answer has to be right.")

from torchlingo.inference import beam_search_decode as library_beam_search from torchlingo.inference import greedy_decode as library_greedy_decode def library_greedy(sentence): """Translate one sentence with the library's greedy decoder.""" ids = src_vocab.encode(sentence, add_special_tokens=True) tokens = library_greedy_decode(model, torch.tensor([ids]).to(device), max_len=20)[0] return tgt_vocab.decode(tokens, skip_special_tokens=True) def library_beam(sentence, beam_size=3): """Translate one sentence with the library's beam search.""" ids = src_vocab.encode(sentence, add_special_tokens=True) tokens = library_beam_search( model, torch.tensor([ids]).to(device), beam_size=beam_size, max_len=20 ) return tgt_vocab.decode(tokens, skip_special_tokens=True) comparisons = 0 for sentence in test_sentences: ours = greedy_decode(model, sentence, src_vocab, tgt_vocab, device) theirs = library_greedy(sentence) assert ours == theirs, f"greedy disagrees on {sentence!r}: {ours!r} vs {theirs!r}" comparisons += 1 for beam_size in (1, 3, 5): ours = beam_search_decode( model, sentence, src_vocab, tgt_vocab, device, beam_size=beam_size ) theirs = library_beam(sentence, beam_size) assert ours == theirs, ( f"beam-{beam_size} disagrees on {sentence!r}: {ours!r} vs {theirs!r}" ) comparisons += 1 print(f"{comparisons} comparisons, notebook and library agree on every one.") print() print("Note what this does and does not show. It shows the two implementations") print("return the same translations for THESE sentences and THIS model. It does") print("not show they are the same algorithm — they are not, as the table above") print("says. Use the library's version when the answer has to be right.")

20 comparisons, notebook and library agree on every one.

Note what this does and does not show. It shows the two implementations
return the same translations for THESE sentences and THIS model. It does
not show they are the same algorithm — they are not, as the table above
says. Use the library's version when the answer has to be right.

Part 5: BLEU Score Evaluation¶

BLEU (Bilingual Evaluation Understudy) measures translation quality by comparing n-gram overlap.

In [12]:

# Install sacrebleu if needed
try:
    from sacrebleu.metrics import BLEU
    print("sacrebleu is installed!")
except ImportError:
    print("Installing sacrebleu...")
    !pip install sacrebleu
    from sacrebleu.metrics import BLEU

# Install sacrebleu if needed try: from sacrebleu.metrics import BLEU print("sacrebleu is installed!") except ImportError: print("Installing sacrebleu...") !pip install sacrebleu from sacrebleu.metrics import BLEU

sacrebleu is installed!

In [13]:

from sacrebleu.metrics import BLEU

# Our test data
sources = [
    "Hello world",
    "Good morning",
    "Thank you",
    "I love you",
]

references = [
    "Hola mundo",
    "Buenos días",
    "Gracias",
    "Te amo",
]

# Generate translations
greedy_translations = [greedy_decode(model, s, src_vocab, tgt_vocab, device) for s in sources]
beam_translations = [beam_search_decode(model, s, src_vocab, tgt_vocab, device) for s in sources]

# Calculate BLEU.
# Standard BLEU is a geometric mean of 1- to 4-gram precisions, so
# 2-3 word phrases score 0 even when they match the reference exactly
# (there are no 4-grams to match!). For this short-phrase demo we use
# bigram BLEU; with real sentence-length data, use the default BLEU().
bleu = BLEU(max_ngram_order=2)

greedy_bleu = bleu.corpus_score(greedy_translations, [references])
beam_bleu = bleu.corpus_score(beam_translations, [references])

print(f"BLEU Scores:")
print(f"  Greedy: {greedy_bleu.score:.2f}")
print(f"  Beam-3: {beam_bleu.score:.2f}")

from sacrebleu.metrics import BLEU # Our test data sources = [ "Hello world", "Good morning", "Thank you", "I love you", ] references = [ "Hola mundo", "Buenos días", "Gracias", "Te amo", ] # Generate translations greedy_translations = [greedy_decode(model, s, src_vocab, tgt_vocab, device) for s in sources] beam_translations = [beam_search_decode(model, s, src_vocab, tgt_vocab, device) for s in sources] # Calculate BLEU. # Standard BLEU is a geometric mean of 1- to 4-gram precisions, so # 2-3 word phrases score 0 even when they match the reference exactly # (there are no 4-grams to match!). For this short-phrase demo we use # bigram BLEU; with real sentence-length data, use the default BLEU(). bleu = BLEU(max_ngram_order=2) greedy_bleu = bleu.corpus_score(greedy_translations, [references]) beam_bleu = bleu.corpus_score(beam_translations, [references]) print(f"BLEU Scores:") print(f" Greedy: {greedy_bleu.score:.2f}") print(f" Beam-3: {beam_bleu.score:.2f}")

BLEU Scores:
  Greedy: 100.00
  Beam-3: 100.00

In [14]:

# Detailed comparison
print(f"{'Source':<20} {'Reference':<20} {'Greedy':<20} {'Beam':<20}")
print("-" * 80)

for src, ref, greedy, beam in zip(sources, references, greedy_translations, beam_translations):
    print(f"{src:<20} {ref:<20} {greedy:<20} {beam:<20}")

# Detailed comparison print(f"{'Source':<20} {'Reference':<20} {'Greedy':<20} {'Beam':<20}") print("-" * 80) for src, ref, greedy, beam in zip(sources, references, greedy_translations, beam_translations): print(f"{src:<20} {ref:<20} {greedy:<20} {beam:<20}")

Source               Reference            Greedy               Beam                
--------------------------------------------------------------------------------
Hello world          Hola mundo           Hola mundo           Hola mundo          
Good morning         Buenos días          Buenos días          Buenos días         
Thank you            Gracias              Gracias              Gracias             
I love you           Te amo               Te amo               Te amo              

Understanding BLEU¶

BLEU measures n-gram precision:

Score	Quality
< 10	Almost unusable
10-20	Gist is clear
20-30	Understandable
30-40	Good quality
40-50	High quality
> 50	Very high quality

⚠️ Don't read anything into the score above. Tutorial 2 trains on twelve phrases and tests on four of those same phrases, so the model has simply memorized them and BLEU comes out at or near 100. That is a sign the mechanics work, not that the model translates well — it has never seen a sentence it wasn't trained on. Real BLEU numbers come from a held-out test set, which is what the table above is describing.

Summary¶

You've learned:

1. Greedy decoding: Fast but can miss better translations
2. Beam search: Explores multiple paths, often better results
3. Length penalty: Prevents beam search from preferring short sequences
4. Tie-breaking: Why torch.topk alone makes decoding device-dependent
5. BLEU score: Standard metric for translation quality
6. Reading a null result: when a sweep tells you about your setup rather than about the parameter you were sweeping

Key Takeaways¶

• Most of the gain is the first beam, and past the peak it reverses. Measured on a real model across several held-out samples, greedy to any beam width is worth roughly 1.2-1.6 BLEU. After that, beam 2 to 3 gains a little, 3 to 5 is flat, and 5 to 10 measurably loses — beam 10 costs about 7x beam 2 to land back where it started. Quality peaks around beam 3 to 5, which is close to the usual default.
• Why widening hurts is the more useful half. Beam search maximizes total log probability, and longer sequences score worse simply for being longer, so a wider search finds shorter output: mean length falls from 12.3 tokens at greedy to 9.7 at beam 10, against references averaging 11.6. The objective and the goal come apart.
• Length penalty is supposed to correct exactly that, and on this model it does not: alpha anywhere from 0.0 to 1.0 is indistinguishable. Sweep it rather than trusting it. Numbers and error bars in Decoding.
• BLEU is useful but not perfect—humans judge translation differently. It is also noisier than it looks: differences of a point or less need error bars before you believe them.
• Use torchlingo.inference in real work. The implementations here are for reading. The library's handle ties deterministically, normalize length during pruning, and have a batched counterpart in torchlingo.inference_fast that produces identical output much faster.

What's Next?¶

• Tutorial 4: Attention and Alignment — what attention learns, on an LSTM
• Tutorial 5: Translating Unseen Sentences — the model this tutorial's measurements needed
• Explore the API Reference for more details
• Read Decoding for the reference-versus-fast split, the full tie-breaking rule, and what the decoding options actually buy
• Learn about SentencePiece for better tokenization
• Try training on a real dataset!