Multilingual

Utilities for handling multilingual data in neural machine translation.

Overview

Multilingual NMT trains a single model to translate between multiple language pairs. This module provides utilities for:

• Adding language tags to source sentences
• Creating bidirectional (EN↔X) training data
• Preparing multilingual datasets from base preprocessed data

Why Multilingual?

Benefit	Description
Transfer learning	Low-resource languages benefit from related high-resource languages
Single model	One model instead of N² models for N languages
Zero-shot	May translate between language pairs not seen during training
Bidirectional	Train both EN→X and X→EN directions simultaneously

Quick Start

from torchlingo.preprocessing.multilingual import (
    add_language_tags,
    preprocess_multilingual,
)

# Add language tags to a DataFrame
df_tagged = add_language_tags(df, tag="<es>")

# Or run the full multilingual pipeline
preprocess_multilingual()  # Uses config defaults

API Reference

add_language_tags

add_language_tags(df: DataFrame, tag: str, src_col: str = None, config: Config = None) -> DataFrame

Prepend a language tag to all source sentences.

Creates a copy of the input DataFrame and prepends a language identifier token to the source column. Used to signal target language to multilingual models.

Parameters:

Name	Type	Description	Default
df	DataFrame	Input DataFrame with source/target columns.	required
tag	str	Language tag to prepend (e.g., '', '').	required
src_col	str	Source column name. Falls back to config.src_col. Defaults to None.	None
config	Config	Configuration object. Defaults to default config.	None

Returns:

Type	Description
DataFrame	pd.DataFrame: Copy of df with language tag prepended to src_col.

Examples:

>>> df = pd.DataFrame({'src': ['hello'], 'tgt': ['hola']})
>>> tagged = add_language_tags(df, '<es>')
>>> tagged['src'].iloc[0]
'<es> hello'

Source code in src/torchlingo/preprocessing/multilingual.py

def add_language_tags(
    df: pd.DataFrame,
    tag: str,
    src_col: str = None,
    config: Config = None,
) -> pd.DataFrame:
    """Prepend a language tag to all source sentences.

    Creates a copy of the input DataFrame and prepends a language identifier
    token to the source column. Used to signal target language to multilingual
    models.

    Args:
        df (pd.DataFrame): Input DataFrame with source/target columns.
        tag (str): Language tag to prepend (e.g., '<en>', '<es>').
        src_col (str, optional): Source column name. Falls back to config.src_col.
            Defaults to None.
        config (Config, optional): Configuration object. Defaults to default config.

    Returns:
        pd.DataFrame: Copy of df with language tag prepended to src_col.

    Examples:
        >>> df = pd.DataFrame({'src': ['hello'], 'tgt': ['hola']})
        >>> tagged = add_language_tags(df, '<es>')
        >>> tagged['src'].iloc[0]
        '<es> hello'
    """
    cfg = config if config is not None else get_default_config()
    src_col = src_col if src_col is not None else cfg.src_col

    df = df.copy()
    df[src_col] = tag + " " + df[src_col].astype(str)
    return df

preprocess_multilingual

preprocess_multilingual(train_file: Path = None, val_file: Path = None, test_file: Path = None, src_col: str = None, tgt_col: str = None, lang_tag_en_to_x: str = None, lang_tag_x_to_en: str = None, seed: int = None, multi_train_file: Path = None, multi_val_file: Path = None, test_en_x_file: Path = None, test_x_en_file: Path = None, data_format: str = None, config: Config = None)

Execute multilingual preprocessing pipeline.

Loads train/val/test splits created by preprocess_base(), creates bidirectional pairs (EN→X and X→EN), tags each direction with appropriate language tags, and saves to multilingual-specific output files.

Requires base preprocessing to have completed (train_file, etc. exist).

Parameters:

Name	Type	Description	Default
train_file	Path	Path to training data. Falls back to config.train_file.	None
val_file	Path	Path to validation data. Falls back to config.val_file.	None
test_file	Path	Path to test data. Falls back to config.test_file.	None
src_col	str	Source column name. Falls back to config.src_col.	None
tgt_col	str	Target column name. Falls back to config.tgt_col.	None
lang_tag_en_to_x	str	Language tag for EN→X direction. Falls back to config.lang_tag_en_to_x.	None
lang_tag_x_to_en	str	Language tag for X→EN direction. Falls back to config.lang_tag_x_to_en.	None
seed	int	Random seed for shuffling. Falls back to config.seed.	None
multi_train_file	Path	Output path for multilingual training data. Falls back to config.multi_train_file.	None
multi_val_file	Path	Output path for multilingual validation data. Falls back to config.multi_val_file.	None
test_en_x_file	Path	Output path for EN→X test set. Falls back to config.test_en_x_file.	None
test_x_en_file	Path	Output path for X→EN test set. Falls back to config.test_x_en_file.	None
data_format	str	Output file format. Falls back to config.data_format.	None
config	Config	Configuration object. Defaults to default config.	None

Creates

• multi_train_file: Combined and shuffled EN→X and X→EN pairs.
• multi_val_file: Combined and shuffled EN→X and X→EN pairs.
• test_en_x_file: Test set for EN→X direction.
• test_x_en_file: Test set for X→EN direction.

Side Effects

Prints status message on completion. Shuffles training/validation using seed for reproducibility. Saves files per specified paths.

Examples:

>>> preprocess_multilingual()
Multilingual preprocessing complete.

Source code in src/torchlingo/preprocessing/multilingual.py

def preprocess_multilingual(
    train_file: Path = None,
    val_file: Path = None,
    test_file: Path = None,
    src_col: str = None,
    tgt_col: str = None,
    lang_tag_en_to_x: str = None,
    lang_tag_x_to_en: str = None,
    seed: int = None,
    multi_train_file: Path = None,
    multi_val_file: Path = None,
    test_en_x_file: Path = None,
    test_x_en_file: Path = None,
    data_format: str = None,
    config: Config = None,
):
    """Execute multilingual preprocessing pipeline.

    Loads train/val/test splits created by preprocess_base(), creates
    bidirectional pairs (EN→X and X→EN), tags each direction with appropriate
    language tags, and saves to multilingual-specific output files.

    Requires base preprocessing to have completed (train_file, etc. exist).

    Args:
        train_file (Path, optional): Path to training data. Falls back to config.train_file.
        val_file (Path, optional): Path to validation data. Falls back to config.val_file.
        test_file (Path, optional): Path to test data. Falls back to config.test_file.
        src_col (str, optional): Source column name. Falls back to config.src_col.
        tgt_col (str, optional): Target column name. Falls back to config.tgt_col.
        lang_tag_en_to_x (str, optional): Language tag for EN→X direction.
            Falls back to config.lang_tag_en_to_x.
        lang_tag_x_to_en (str, optional): Language tag for X→EN direction.
            Falls back to config.lang_tag_x_to_en.
        seed (int, optional): Random seed for shuffling. Falls back to config.seed.
        multi_train_file (Path, optional): Output path for multilingual training data.
            Falls back to config.multi_train_file.
        multi_val_file (Path, optional): Output path for multilingual validation data.
            Falls back to config.multi_val_file.
        test_en_x_file (Path, optional): Output path for EN→X test set.
            Falls back to config.test_en_x_file.
        test_x_en_file (Path, optional): Output path for X→EN test set.
            Falls back to config.test_x_en_file.
        data_format (str, optional): Output file format. Falls back to config.data_format.
        config (Config, optional): Configuration object. Defaults to default config.

    Creates:
        - multi_train_file: Combined and shuffled EN→X and X→EN pairs.
        - multi_val_file: Combined and shuffled EN→X and X→EN pairs.
        - test_en_x_file: Test set for EN→X direction.
        - test_x_en_file: Test set for X→EN direction.

    Side Effects:
        Prints status message on completion. Shuffles training/validation
        using seed for reproducibility. Saves files per specified paths.

    Examples:
        >>> preprocess_multilingual()
        Multilingual preprocessing complete.
    """
    cfg = config if config is not None else get_default_config()
    train_file = train_file if train_file is not None else cfg.train_file
    val_file = val_file if val_file is not None else cfg.val_file
    test_file = test_file if test_file is not None else cfg.test_file
    src_col = src_col if src_col is not None else cfg.src_col
    tgt_col = tgt_col if tgt_col is not None else cfg.tgt_col
    lang_tag_en_to_x = (
        lang_tag_en_to_x if lang_tag_en_to_x is not None else cfg.lang_tag_en_to_x
    )
    lang_tag_x_to_en = (
        lang_tag_x_to_en if lang_tag_x_to_en is not None else cfg.lang_tag_x_to_en
    )
    seed = seed if seed is not None else cfg.seed
    multi_train_file = (
        multi_train_file if multi_train_file is not None else cfg.multi_train_file
    )
    multi_val_file = (
        multi_val_file if multi_val_file is not None else cfg.multi_val_file
    )
    test_en_x_file = (
        test_en_x_file if test_en_x_file is not None else cfg.test_en_x_file
    )
    test_x_en_file = (
        test_x_en_file if test_x_en_file is not None else cfg.test_x_en_file
    )
    data_format = data_format if data_format is not None else cfg.data_format

    if not train_file.exists():
        print("Run base preprocessing first to generate train/val/test.")
        return
    train_df = load_data(train_file)
    val_df = load_data(val_file)
    test_df = load_data(test_file)
    train_en_x = train_df.copy()
    val_en_x = val_df.copy()
    test_en_x = test_df.copy()
    train_x_en = train_df.rename(columns={src_col: tgt_col, tgt_col: src_col})
    val_x_en = val_df.rename(columns={src_col: tgt_col, tgt_col: src_col})
    test_x_en = test_df.rename(columns={src_col: tgt_col, tgt_col: src_col})
    train_en_x = add_language_tags(train_en_x, lang_tag_en_to_x, src_col=src_col)
    train_x_en = add_language_tags(train_x_en, lang_tag_x_to_en, src_col=src_col)
    val_en_x = add_language_tags(val_en_x, lang_tag_en_to_x, src_col=src_col)
    val_x_en = add_language_tags(val_x_en, lang_tag_x_to_en, src_col=src_col)
    test_en_x = add_language_tags(test_en_x, lang_tag_en_to_x, src_col=src_col)
    test_x_en = add_language_tags(test_x_en, lang_tag_x_to_en, src_col=src_col)
    combined_train = (
        pd.concat([train_en_x, train_x_en])
        .sample(frac=1, random_state=seed)
        .reset_index(drop=True)
    )
    combined_val = (
        pd.concat([val_en_x, val_x_en])
        .sample(frac=1, random_state=seed)
        .reset_index(drop=True)
    )
    save_data(combined_train, multi_train_file, data_format)
    save_data(combined_val, multi_val_file, data_format)
    save_data(test_en_x, test_en_x_file, data_format)
    save_data(test_x_en, test_x_en_file, data_format)
    print("Multilingual preprocessing complete.")

Language Tag Format

Language tags are prepended to the source sentence to indicate the target language:

Source: "Hello world" → "<es> Hello world"
Target: "Hola mundo"

The model learns to associate <es> with Spanish translations.

How It Works

from torchlingo.preprocessing.multilingual import add_language_tags
import pandas as pd

# Original data
df = pd.DataFrame({
    'src': ['Hello', 'Goodbye'],
    'tgt': ['Hola', 'Adiós']
})

# Add Spanish target tag
df_tagged = add_language_tags(df, tag='<es>')
print(df_tagged['src'][0])
# '<es> Hello'

Examples

Basic Usage - Adding Tags

from torchlingo.preprocessing.multilingual import add_language_tags
from torchlingo.preprocessing.base import load_data

# Load English-Spanish data
df = load_data("data/train.tsv")

# Add target language tag
df_tagged = add_language_tags(df, tag="<es>")

print(df_tagged.head())
#                   src            tgt
# 0  <es> Hello world   Hola mundo
# 1  <es> Good morning  Buenos días

Full Multilingual Pipeline

The preprocess_multilingual function handles the complete bidirectional data preparation:

from torchlingo.preprocessing.multilingual import preprocess_multilingual
from torchlingo.config import Config

# Configure your paths and tags
config = Config(
    train_file="data/train.tsv",
    val_file="data/val.tsv",
    test_file="data/test.tsv",
    lang_tag_en_to_x="<es>",   # Tag for English→Spanish
    lang_tag_x_to_en="<en>",   # Tag for Spanish→English
    multi_train_file="data/multi_train.tsv",
    multi_val_file="data/multi_val.tsv",
    test_en_x_file="data/test_en_es.tsv",
    test_x_en_file="data/test_es_en.tsv",
)

# Run the pipeline
preprocess_multilingual(config=config)

This will:

1. Load your base train/val/test splits
2. Create EN→X copies (tagged with <es>)
3. Create X→EN copies (columns swapped, tagged with <en>)
4. Combine and shuffle training/validation data
5. Save separate test sets for each direction

Understanding Bidirectional Data

# Input data (EN→ES):
# src: "Hello"  → tgt: "Hola"

# After preprocess_multilingual:

# Direction 1 (EN→ES):
# src: "<es> Hello"  → tgt: "Hola"

# Direction 2 (ES→EN):  
# src: "<en> Hola"   → tgt: "Hello"

Both directions are combined into the training data, shuffled together.

Configuration Options

The multilingual pipeline uses these configuration options:

Option	Description	Default
train_file	Base training data path	data/train.tsv
val_file	Base validation data path	data/val.tsv
test_file	Base test data path	data/test.tsv
src_col	Source column name	src
tgt_col	Target column name	tgt
lang_tag_en_to_x	Tag for EN→X direction	<es>
lang_tag_x_to_en	Tag for X→EN direction	<en>
multi_train_file	Output multilingual train path	data/multi_train.tsv
multi_val_file	Output multilingual val path	data/multi_val.tsv
test_en_x_file	Output EN→X test path	data/test_en_x.tsv
test_x_en_file	Output X→EN test path	data/test_x_en.tsv

Best Practices

Run Base Preprocessing First

The multilingual pipeline expects that base preprocessing has already created train_file, val_file, and test_file. Run preprocess_base() first!

Consistent Tags

Use consistent language tags throughout your project. Common conventions:

• <lang> - Simple: <en>, <es>, <fr>
• >>lang<< - OPUS style: >>en<<, >>es<<
• [lang] - Bracketed: [en], [es]

Separate Test Sets

The pipeline creates separate test sets for each direction. This lets you evaluate EN→X and X→EN performance independently.

Common Workflow

from torchlingo.preprocessing.base import preprocess_base
from torchlingo.preprocessing.multilingual import preprocess_multilingual
from torchlingo.config import Config, set_default_config

# 1. Set up configuration
config = Config(
    src_file="data/english.txt",
    tgt_file="data/spanish.txt",
    lang_tag_en_to_x="<es>",
    lang_tag_x_to_en="<en>",
)
set_default_config(config)

# 2. Base preprocessing (creates train/val/test)
preprocess_base()

# 3. Multilingual preprocessing (creates bidirectional data)
preprocess_multilingual()

# Now you have:
# - data/multi_train.tsv (bidirectional training)
# - data/multi_val.tsv (bidirectional validation)
# - data/test_en_x.tsv (EN→ES test)
# - data/test_x_en.tsv (ES→EN test)