Vocabulary¶

Vocabulary classes for encoding and decoding text in neural machine translation. All implementations share the BaseVocab interface so they can be swapped without changing calling code.

Overview¶

TorchLingo provides a common interface plus multiple implementations:

Class	Use Case
`BaseVocab`	Abstract interface for encoding/decoding
`SimpleVocab`	Simple whitespace tokenization, good for small datasets
`SentencePieceVocab`	Subword tokenization, handles rare words better
`MeCabVocab`	Japanese morphological analysis (requires `fugashi`)
`JiebaVocab`	Chinese word segmentation (requires `jieba`)

All vocab classes implement the BaseVocab contract, so any function annotated with BaseVocab can accept any implementation.

Quick Start¶

from torchlingo.data_processing import (
    BaseVocab,
    SimpleVocab,
    SentencePieceVocab,
    MeCabVocab,
    JiebaVocab,
)

# Simple vocabulary
vocab: BaseVocab = SimpleVocab()
vocab.build_vocab(["Hello world", "How are you"])
indices = vocab.encode("Hello world", add_special_tokens=True)
# [2, 4, 5, 3]  (SOS, Hello, world, EOS)

# SentencePiece vocabulary
sp_vocab = SentencePieceVocab("models/sp.model")
indices = sp_vocab.encode("Hello world", add_special_tokens=True)

# Japanese vocabulary (requires: pip install torchlingo[japanese])
jp_vocab = MeCabVocab(min_freq=1)
jp_vocab.build_vocab(["私は学生です", "彼は先生です"])
indices = jp_vocab.encode("私は学生です")

# Chinese vocabulary (requires: pip install torchlingo[chinese])
zh_vocab = JiebaVocab(min_freq=1)
zh_vocab.build_vocab(["我是学生", "他是老师"])
indices = zh_vocab.encode("我是学生")

API Reference¶

BaseVocab¶

BaseVocab ¶

BaseVocab(*, pad_token: Optional[str] = None, unk_token: Optional[str] = None, sos_token: Optional[str] = None, eos_token: Optional[str] = None, pad_idx: Optional[int] = None, unk_idx: Optional[int] = None, sos_idx: Optional[int] = None, eos_idx: Optional[int] = None, config: Optional[Config] = None)

Bases: ABC

Abstract base class describing the vocabulary contract.

Subclasses must implement the abstract methods below. The base class centralizes handling of special tokens (PAD/UNK/SOS/EOS) and their default indices so that concrete implementations behave consistently.

Implementations are expected to be lightweight and to focus on token splitting/merging (encode/decode) and token<->index conversions. The SimpleVocab stores an in-memory mapping and supports building from raw sentences; SentencePieceVocab wraps a pre-trained SentencePiece model and therefore does not implement build_vocab.

Examples:

>>> from torchlingo.data_processing import BaseVocab, SimpleVocab
>>> v: BaseVocab = SimpleVocab()
>>> v.build_vocab(["a b", "a c"])  # only available for SimpleVocab
>>> v.encode("a b")

Source code in src/torchlingo/data_processing/vocab.py

def __init__(
    self,
    *,
    pad_token: Optional[str] = None,
    unk_token: Optional[str] = None,
    sos_token: Optional[str] = None,
    eos_token: Optional[str] = None,
    pad_idx: Optional[int] = None,
    unk_idx: Optional[int] = None,
    sos_idx: Optional[int] = None,
    eos_idx: Optional[int] = None,
    config: Optional[Config] = None,
) -> None:
    cfg = config if config is not None else get_default_config()

    self.pad_token = pad_token if pad_token is not None else cfg.pad_token
    self.unk_token = unk_token if unk_token is not None else cfg.unk_token
    self.sos_token = sos_token if sos_token is not None else cfg.sos_token
    self.eos_token = eos_token if eos_token is not None else cfg.eos_token

    self.pad_idx = pad_idx if pad_idx is not None else cfg.pad_idx
    self.unk_idx = unk_idx if unk_idx is not None else cfg.unk_idx
    self.sos_idx = sos_idx if sos_idx is not None else cfg.sos_idx
    self.eos_idx = eos_idx if eos_idx is not None else cfg.eos_idx

len `abstractmethod` ¶

__len__() -> int

Return the vocabulary size (number of token ids available).

The returned value must include any reserved special tokens (PAD/UNK/ SOS/EOS) so that consumers can size embedding layers and related structures using len(vocab).

Returns:

Name	Type	Description
`int`	`int`	Total number of tokens in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

@abstractmethod
def __len__(self) -> int:
    """Return the vocabulary size (number of token ids available).

    The returned value must include any reserved special tokens (PAD/UNK/
    SOS/EOS) so that consumers can size embedding layers and related
    structures using `len(vocab)`.

    Returns:
        int: Total number of tokens in the vocabulary.
    """

build_vocab `abstractmethod` ¶

build_vocab(sentences: Sequence[str]) -> None

Populate internal structures from raw sentences.

Not all implementations are required to support building a vocabulary from raw text (for example, SentencePieceVocab loads a pre-trained model and will raise). Implementations that do support building should accept an iterable of raw (untokenized or tokenized) sentence strings.

Parameters:

Name	Type	Description	Default
`sentences`	`Sequence[str]`	Iterable of raw sentence strings.	required

Source code in src/torchlingo/data_processing/vocab.py

@abstractmethod
def build_vocab(self, sentences: Sequence[str]) -> None:
    """Populate internal structures from raw sentences.

    Not all implementations are required to support building a vocabulary
    from raw text (for example, `SentencePieceVocab` loads a pre-trained
    model and will raise). Implementations that *do* support building
    should accept an iterable of raw (untokenized or tokenized) sentence
    strings.

    Args:
        sentences (Sequence[str]): Iterable of raw sentence strings.
    """

token_to_idx `abstractmethod` ¶

token_to_idx(token: str) -> int

Return the integer id for token.

If token is not present in the vocabulary, implementations should return the configured unknown index (unk_idx).

Parameters:

Name	Type	Description	Default
`token`	`str`	Token string to look up.	required

Returns:

Name	Type	Description
`int`	`int`	Integer index for the token.

Source code in src/torchlingo/data_processing/vocab.py

@abstractmethod
def token_to_idx(self, token: str) -> int:
    """Return the integer id for `token`.

    If `token` is not present in the vocabulary, implementations should
    return the configured unknown index (`unk_idx`).

    Args:
        token (str): Token string to look up.

    Returns:
        int: Integer index for the token.
    """

idx_to_token `abstractmethod` ¶

idx_to_token(idx: int) -> str

Return the token string for idx.

Unknown indices should map to the configured unknown token string (unk_token).

Parameters:

Name	Type	Description	Default
`idx`	`int`	Integer index to look up.	required

Returns:

Name	Type	Description
`str`	`str`	Token string corresponding to the index.

Source code in src/torchlingo/data_processing/vocab.py

@abstractmethod
def idx_to_token(self, idx: int) -> str:
    """Return the token string for `idx`.

    Unknown indices should map to the configured unknown token string
    (`unk_token`).

    Args:
        idx (int): Integer index to look up.

    Returns:
        str: Token string corresponding to the index.
    """

tokens_to_indices ¶

tokens_to_indices(tokens: Sequence[str]) -> List[int]

Helper: map a sequence of tokens to their indices.

Implemented in terms of token_to_idx to guarantee consistent UNK handling across vocab implementations.

Parameters:

Name	Type	Description	Default
`tokens`	`Sequence[str]`	List of token strings.	required

Returns:

Type	Description
`List[int]`	List[int]: List of corresponding token indices.

Source code in src/torchlingo/data_processing/vocab.py

def tokens_to_indices(self, tokens: Sequence[str]) -> List[int]:
    """Helper: map a sequence of tokens to their indices.

    Implemented in terms of `token_to_idx` to guarantee consistent UNK
    handling across vocab implementations.

    Args:
        tokens (Sequence[str]): List of token strings.

    Returns:
        List[int]: List of corresponding token indices.
    """

    return [self.token_to_idx(token) for token in tokens]

indices_to_tokens ¶

indices_to_tokens(indices: Sequence[int]) -> List[str]

Helper: map a sequence of indices to their token strings.

Implemented in terms of idx_to_token and preserves the order of the input indices.

Parameters:

Name	Type	Description	Default
`indices`	`Sequence[int]`	List of token indices.	required

Returns:

Type	Description
`List[str]`	List[str]: List of corresponding token strings.

Source code in src/torchlingo/data_processing/vocab.py

def indices_to_tokens(self, indices: Sequence[int]) -> List[str]:
    """Helper: map a sequence of indices to their token strings.

    Implemented in terms of `idx_to_token` and preserves the order of the
    input indices.

    Args:
        indices (Sequence[int]): List of token indices.

    Returns:
        List[str]: List of corresponding token strings.
    """

    return [self.idx_to_token(idx) for idx in indices]

encode `abstractmethod` ¶

encode(sentence: str, add_special_tokens: bool = True) -> List[int]

Convert a raw sentence to a list of token ids.

Parameters:

Name	Type	Description	Default
`sentence`	`str`	Raw input string. For simple vocabularies this is expected to be an untokenized string; for SentencePiece it may be preprocessed as appropriate for the model.	required
`add_special_tokens`	`bool`	If True, the encoder should prepend `sos_idx` and append `eos_idx` to the returned id sequence. Defaults to True.	`True`

Returns:

Type	Description
`List[int]`	List[int]: A list of integer token ids.

Source code in src/torchlingo/data_processing/vocab.py

@abstractmethod
def encode(self, sentence: str, add_special_tokens: bool = True) -> List[int]:
    """Convert a raw sentence to a list of token ids.

    Args:
        sentence (str): Raw input string. For simple vocabularies this is
            expected to be an untokenized string; for SentencePiece it may
            be preprocessed as appropriate for the model.
        add_special_tokens (bool, optional): If True, the encoder should
            prepend `sos_idx` and append `eos_idx` to the returned id
            sequence. Defaults to True.

    Returns:
        List[int]: A list of integer token ids.
    """

decode `abstractmethod` ¶

decode(indices: IndexInput, skip_special_tokens: bool = True) -> DecodedOutput

Decode integer ids back to human-readable text.

The method must accept either a flat 1D sequence of ids, a 2D batched sequence (list of lists) or a 1/2-D torch.Tensor. When a batch is provided the method should return a list of decoded strings; for a 1-D input it should return a single string.

Parameters:

Name	Type	Description	Default
`indices`	`IndexInput`	1-D or 2-D sequence of token ids or a tensor.	required
`skip_special_tokens`	`bool`	When True, remove any special PAD/SOS/EOS/UNK tokens from the decoded output (useful for displaying model output). Defaults to True.	`True`

Returns:

Name	Type	Description
`DecodedOutput`	`DecodedOutput`	Decoded text (str) or list of strings.

Source code in src/torchlingo/data_processing/vocab.py

@abstractmethod
def decode(
    self, indices: IndexInput, skip_special_tokens: bool = True
) -> DecodedOutput:
    """Decode integer ids back to human-readable text.

    The method must accept either a flat 1D sequence of ids, a 2D batched
    sequence (list of lists) or a 1/2-D `torch.Tensor`. When a batch is
    provided the method should return a list of decoded strings; for a 1-D
    input it should return a single string.

    Args:
        indices (IndexInput): 1-D or 2-D sequence of token ids or a tensor.
        skip_special_tokens (bool, optional): When True, remove any special
            PAD/SOS/EOS/UNK tokens from the decoded output (useful for
            displaying model output). Defaults to True.

    Returns:
        DecodedOutput: Decoded text (str) or list of strings.
    """

SimpleVocab¶

SimpleVocab ¶

SimpleVocab(min_freq: Optional[int] = None, *, pad_token: Optional[str] = None, unk_token: Optional[str] = None, sos_token: Optional[str] = None, eos_token: Optional[str] = None, pad_idx: Optional[int] = None, unk_idx: Optional[int] = None, sos_idx: Optional[int] = None, eos_idx: Optional[int] = None, config: Optional[Config] = None)

Bases: BaseVocab

Simple whitespace-tokenized vocabulary with frequency-based filtering.

Builds a vocabulary by splitting raw text on whitespace and counting token frequencies. Only tokens appearing at least min_freq times are included. Special tokens (PAD, UNK, SOS, EOS) are always reserved and included.

This vocabulary is suitable for small datasets or when you have limited preprocessing control. For production use or languages with complex word boundaries, consider SentencePieceVocab.

Parameters:

Name	Type	Description	Default
`min_freq`	`int`	Minimum frequency threshold for including a token in the vocabulary. Tokens appearing fewer than min_freq times are mapped to the UNK token index. If None, uses value from config. Defaults to None.	`None`
`pad_token`	`str`	Padding token string. If None, uses value from config. Defaults to None.	`None`
`unk_token`	`str`	Unknown token string. If None, uses value from config. Defaults to None.	`None`
`sos_token`	`str`	Start-of-sequence token string. If None, uses value from config. Defaults to None.	`None`
`eos_token`	`str`	End-of-sequence token string. If None, uses value from config. Defaults to None.	`None`
`pad_idx`	`int`	Padding token index. If None, uses value from config. Defaults to None.	`None`
`unk_idx`	`int`	Unknown token index. If None, uses value from config. Defaults to None.	`None`
`sos_idx`	`int`	Start-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`eos_idx`	`int`	End-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`config`	`Config`	Configuration object to use for default values. If None, uses get_default_config(). Defaults to None.	`None`

Attributes:

Name	Type	Description
`token2idx`	`dict`	Mapping from token strings to integer indices.
`idx2token`	`dict`	Mapping from integer indices to token strings.
`token_freqs`	`dict`	Frequency count for each observed token.
`min_freq`	`int`	Minimum frequency threshold.
`pad_token`	`str`	Padding token string.
`unk_token`	`str`	Unknown token string.
`sos_token`	`str`	Start-of-sequence token string.
`eos_token`	`str`	End-of-sequence token string.
`pad_idx`	`int`	Padding token index.
`unk_idx`	`int`	Unknown token index.
`sos_idx`	`int`	Start-of-sequence token index.
`eos_idx`	`int`	End-of-sequence token index.

Source code in src/torchlingo/data_processing/vocab.py

def __init__(
    self,
    min_freq: Optional[int] = None,
    *,
    pad_token: Optional[str] = None,
    unk_token: Optional[str] = None,
    sos_token: Optional[str] = None,
    eos_token: Optional[str] = None,
    pad_idx: Optional[int] = None,
    unk_idx: Optional[int] = None,
    sos_idx: Optional[int] = None,
    eos_idx: Optional[int] = None,
    config: Optional[Config] = None,
) -> None:
    cfg = config if config is not None else get_default_config()
    super().__init__(
        pad_token=pad_token,
        unk_token=unk_token,
        sos_token=sos_token,
        eos_token=eos_token,
        pad_idx=pad_idx,
        unk_idx=unk_idx,
        sos_idx=sos_idx,
        eos_idx=eos_idx,
        config=cfg,
    )

    self.min_freq = min_freq if min_freq is not None else cfg.min_freq
    self.token2idx = {
        self.pad_token: self.pad_idx,
        self.unk_token: self.unk_idx,
        self.sos_token: self.sos_idx,
        self.eos_token: self.eos_idx,
    }
    self.idx2token = {idx: tok for tok, idx in self.token2idx.items()}
    self.token_freqs: dict[str, int] = {}

build_vocab ¶

build_vocab(sentences: Sequence[str]) -> None

Build the vocabulary from a list of raw sentences.

Counts token frequencies by splitting each sentence on whitespace. Adds tokens with frequency >= min_freq to the vocabulary, skipping special tokens that are pre-initialized.

Parameters:

Name	Type	Description	Default
`sentences`	`list[str]`	List of raw sentences to build vocabulary from. Each sentence is assumed to be whitespace-separated tokens.	required

Notes

Modifies self.token_freqs and self.token2idx in place.
Special tokens (PAD, UNK, SOS, EOS) are not overwritten.
Tokens are assigned indices in order of appearance in the iteration.

Source code in src/torchlingo/data_processing/vocab.py

def build_vocab(self, sentences: Sequence[str]) -> None:
    """Build the vocabulary from a list of raw sentences.

    Counts token frequencies by splitting each sentence on whitespace.
    Adds tokens with frequency >= min_freq to the vocabulary, skipping
    special tokens that are pre-initialized.

    Args:
        sentences (list[str]): List of raw sentences to build vocabulary from.
            Each sentence is assumed to be whitespace-separated tokens.

    Notes:
        - Modifies self.token_freqs and self.token2idx in place.
        - Special tokens (PAD, UNK, SOS, EOS) are not overwritten.
        - Tokens are assigned indices in order of appearance in the iteration.
    """

    for sentence in sentences:
        for token in sentence.split():
            self.token_freqs[token] = self.token_freqs.get(token, 0) + 1

    for token, freq in self.token_freqs.items():
        if freq >= self.min_freq and token not in self.token2idx:
            idx = len(self.token2idx)
            self.token2idx[token] = idx
            self.idx2token[idx] = token

len ¶

__len__() -> int

Return the total vocabulary size.

Returns:

Name	Type	Description
`int`	`int`	Number of unique tokens in the vocabulary, including special tokens.

Source code in src/torchlingo/data_processing/vocab.py

def __len__(self) -> int:
    """Return the total vocabulary size.

    Returns:
        int: Number of unique tokens in the vocabulary, including special tokens.
    """

    return len(self.token2idx)

token_to_idx ¶

token_to_idx(token: str) -> int

Convert a single token string to its vocabulary index.

Parameters:

Name	Type	Description	Default
`token`	`str`	The token to convert.	required

Returns:

Name	Type	Description
`int`	`int`	The vocabulary index of the token. Returns unk_idx if the token is not in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

def token_to_idx(self, token: str) -> int:
    """Convert a single token string to its vocabulary index.

    Args:
        token (str): The token to convert.

    Returns:
        int: The vocabulary index of the token. Returns unk_idx if the token
            is not in the vocabulary.
    """

    return self.token2idx.get(token, self.unk_idx)

idx_to_token ¶

idx_to_token(idx: int) -> str

Convert a vocabulary index to its token string.

Parameters:

Name	Type	Description	Default
`idx`	`int`	The vocabulary index.	required

Returns:

Name	Type	Description
`str`	`str`	The token string corresponding to the index. Returns unk_token if the index is not in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

def idx_to_token(self, idx: int) -> str:
    """Convert a vocabulary index to its token string.

    Args:
        idx (int): The vocabulary index.

    Returns:
        str: The token string corresponding to the index. Returns unk_token
            if the index is not in the vocabulary.
    """

    return self.idx2token.get(idx, self.unk_token)

encode ¶

encode(sentence: str, add_special_tokens: bool = True) -> List[int]

Encode a sentence to a list of vocabulary indices.

Splits the sentence on whitespace and converts each token to its vocabulary index. Optionally prepends SOS (start-of-sequence) and appends EOS (end-of-sequence) indices.

Parameters:

Name	Type	Description	Default
`sentence`	`str`	Raw sentence string, whitespace-separated tokens.	required
`add_special_tokens`	`bool`	If True, prepend SOS_IDX and append EOS_IDX to the sequence. Defaults to True.	`True`

Returns:

Type	Description
`List[int]`	list[int]: Encoded sequence of vocabulary indices. Shape is (len(tokens) + 2,) if add_special_tokens is True, else (len(tokens),).

Examples:

>>> vocab = SimpleVocab(min_freq=1)
>>> vocab.build_vocab(["hello world", "world peace"])
>>> indices = vocab.encode("hello world", add_special_tokens=False)
>>> len(indices)  # 2 tokens
2
>>> indices_with_special = vocab.encode("hello world")
>>> len(indices_with_special)  # 2 tokens + SOS + EOS
4

Source code in src/torchlingo/data_processing/vocab.py

def encode(self, sentence: str, add_special_tokens: bool = True) -> List[int]:
    """Encode a sentence to a list of vocabulary indices.

    Splits the sentence on whitespace and converts each token to its
    vocabulary index. Optionally prepends SOS (start-of-sequence) and
    appends EOS (end-of-sequence) indices.

    Args:
        sentence (str): Raw sentence string, whitespace-separated tokens.
        add_special_tokens (bool, optional): If True, prepend SOS_IDX and
            append EOS_IDX to the sequence. Defaults to True.

    Returns:
        list[int]: Encoded sequence of vocabulary indices. Shape is
            (len(tokens) + 2,) if add_special_tokens is True, else
            (len(tokens),).

    Examples:
        >>> vocab = SimpleVocab(min_freq=1)
        >>> vocab.build_vocab(["hello world", "world peace"])
        >>> indices = vocab.encode("hello world", add_special_tokens=False)
        >>> len(indices)  # 2 tokens
        2
        >>> indices_with_special = vocab.encode("hello world")
        >>> len(indices_with_special)  # 2 tokens + SOS + EOS
        4
    """

    tokens = sentence.split()
    indices = self.tokens_to_indices(tokens)
    if add_special_tokens:
        indices = [self.sos_idx] + indices + [self.eos_idx]
    return indices

decode ¶

decode(indices: IndexInput, skip_special_tokens: bool = True) -> DecodedOutput

Decode indices back to text (supports batched inputs).

Accepts a flat 1D sequence of indices or a batched 2D structure (list[list[int]] or tensor of shape [batch, seq]). Batched inputs return a list of decoded strings; flat inputs return a single string.

Parameters:

Name	Type	Description	Default
`indices`	`Union[Sequence[int], Sequence[Sequence[int]]`	1D indices, 2D nested indices, or tensor.	required
`skip_special_tokens`	`bool`	If True, drop PAD/SOS/EOS/UNK tokens.	`True`

Returns:

Type	Description
`DecodedOutput`	str \| list[str]: Decoded text for each sequence.

Examples:

>>> vocab = SimpleVocab(min_freq=1)
>>> vocab.build_vocab(["hello world"])
>>> indices = vocab.encode("hello world")
>>> decoded = vocab.decode(indices)
>>> decoded
'hello world'

Source code in src/torchlingo/data_processing/vocab.py

def decode(
    self, indices: IndexInput, skip_special_tokens: bool = True
) -> DecodedOutput:
    """Decode indices back to text (supports batched inputs).

    Accepts a flat 1D sequence of indices or a batched 2D structure
    (list[list[int]] or tensor of shape [batch, seq]). Batched inputs
    return a list of decoded strings; flat inputs return a single string.

    Args:
        indices (Union[Sequence[int], Sequence[Sequence[int]]): 1D indices, 2D nested indices, or tensor.
        skip_special_tokens (bool, optional): If True, drop PAD/SOS/EOS/UNK tokens.

    Returns:
        str | list[str]: Decoded text for each sequence.

    Examples:
        >>> vocab = SimpleVocab(min_freq=1)
        >>> vocab.build_vocab(["hello world"])
        >>> indices = vocab.encode("hello world")
        >>> decoded = vocab.decode(indices)
        >>> decoded
        'hello world'
    """

    normalized = self._coerce_indices(indices)

    if isinstance(normalized, Sequence) and self._is_batch(normalized):
        return [
            self.decode(list(seq), skip_special_tokens=skip_special_tokens)  # type: ignore[list-item]
            for seq in normalized
        ]

    flat: List[int] = list(normalized)  # type: ignore[list-item]
    tokens = self.indices_to_tokens(flat)

    if skip_special_tokens:
        specials = {self.pad_token, self.unk_token, self.sos_token, self.eos_token}
        tokens = [tok for tok in tokens if tok not in specials]

    return " ".join(tokens)

SentencePieceVocab¶

SentencePieceVocab ¶

SentencePieceVocab(model_path: str, pad_idx: Optional[int] = None, unk_idx: Optional[int] = None, sos_idx: Optional[int] = None, eos_idx: Optional[int] = None, config: Optional[Config] = None)

Bases: BaseVocab

Vocabulary wrapper for pre-trained SentencePiece subword tokenization models.

Loads and uses a SentencePiece model for encoding and decoding text. SentencePiece handles language-specific preprocessing, rare characters, and provides statistically-driven subword segmentation.

Parameters:

Name	Type	Description	Default
`model_path`	`str`	Path to a SentencePiece model file (.model).	required
`pad_idx`	`int`	Padding token index. If None, uses value from config. Defaults to None.	`None`
`unk_idx`	`int`	Unknown token index. If None, uses value from config. Defaults to None.	`None`
`sos_idx`	`int`	Start-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`eos_idx`	`int`	End-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`config`	`Config`	Configuration object to use for default values. If None, uses get_default_config(). Defaults to None.	`None`

Raises:

Type	Description
`FileNotFoundError`	If the model file does not exist.
`RuntimeError`	If the model cannot be loaded by SentencePiece.

Attributes:

Name	Type	Description
`sp`	`SentencePieceProcessor`	The loaded SentencePiece model instance.
`pad_idx`	`int`	Padding token index.
`unk_idx`	`int`	Unknown token index.
`sos_idx`	`int`	Start-of-sequence token index.
`eos_idx`	`int`	End-of-sequence token index.

Source code in src/torchlingo/data_processing/vocab.py

def __init__(
    self,
    model_path: str,
    pad_idx: Optional[int] = None,
    unk_idx: Optional[int] = None,
    sos_idx: Optional[int] = None,
    eos_idx: Optional[int] = None,
    config: Optional[Config] = None,
) -> None:
    cfg = config if config is not None else get_default_config()
    super().__init__(
        pad_idx=pad_idx,
        unk_idx=unk_idx,
        sos_idx=sos_idx,
        eos_idx=eos_idx,
        pad_token=cfg.pad_token,
        unk_token=cfg.unk_token,
        sos_token=cfg.sos_token,
        eos_token=cfg.eos_token,
        config=cfg,
    )

    # Load sentencepiece processor; the load() call returns False when the
    # model could not be found/loaded which we surface as FileNotFoundError
    # to make failures explicit and easy to debug in upstream code.
    self.sp: spm.SentencePieceProcessor = spm.SentencePieceProcessor()
    loaded = self.sp.load(model_path)
    if not loaded:
        raise FileNotFoundError(f"SentencePiece model not found: {model_path}")

len ¶

__len__() -> int

Return the vocabulary size (number of pieces).

Returns:

Name	Type	Description
`int`	`int`	Total number of subword pieces in the SentencePiece model.

Source code in src/torchlingo/data_processing/vocab.py

def __len__(self) -> int:
    """Return the vocabulary size (number of pieces).

    Returns:
        int: Total number of subword pieces in the SentencePiece model.
    """

    return self.sp.get_piece_size()

encode ¶

encode(sentence: str, add_special_tokens: bool = True) -> List[int]

Encode a sentence using the SentencePiece model.

Parameters:

Name	Type	Description	Default
`sentence`	`str`	Raw sentence to encode.	required
`add_special_tokens`	`bool`	If True, prepend sos_idx and append eos_idx. Defaults to True.	`True`

Returns:

Type	Description
`List[int]`	list[int]: List of subword piece indices. Shape is (num_pieces + 2,) if add_special_tokens is True, else (num_pieces,).

Source code in src/torchlingo/data_processing/vocab.py

def encode(self, sentence: str, add_special_tokens: bool = True) -> List[int]:
    """Encode a sentence using the SentencePiece model.

    Args:
        sentence (str): Raw sentence to encode.
        add_special_tokens (bool, optional): If True, prepend sos_idx and
            append eos_idx. Defaults to True.

    Returns:
        list[int]: List of subword piece indices. Shape is (num_pieces + 2,)
            if add_special_tokens is True, else (num_pieces,).
    """

    pieces = self.sp.encode(sentence, out_type=int)
    if add_special_tokens:
        return [self.sos_idx] + pieces + [self.eos_idx]
    return list(pieces)

decode ¶

decode(indices: IndexInput, skip_special_tokens: bool = True) -> DecodedOutput

Decode piece indices back to a sentence (supports batched inputs).

Accepts a flat 1D sequence or a batched 2D structure (list[list[int]] or tensor). Batched inputs return a list of decoded strings; flat inputs return a single string.

Parameters:

Name	Type	Description	Default
`indices`	`(list[int], list[list[int]], tensor)`	1D/2D indices or tensor.	required
`skip_special_tokens`	`bool`	If True, drop PAD/SOS/EOS/UNK tokens before decoding. Defaults to True.	`True`

Returns:

Name	Type	Description
`text`	`(str, list[str])`	Decoded string for flat inputs or list of strings for batched inputs.

Source code in src/torchlingo/data_processing/vocab.py

def decode(
    self, indices: IndexInput, skip_special_tokens: bool = True
) -> DecodedOutput:
    """Decode piece indices back to a sentence (supports batched inputs).

    Accepts a flat 1D sequence or a batched 2D structure (list[list[int]]
    or tensor). Batched inputs return a list of decoded strings; flat inputs
    return a single string.

    Args:
        indices (list[int], list[list[int]], tensor): 1D/2D indices or tensor.
        skip_special_tokens (bool, optional): If True, drop PAD/SOS/EOS/UNK
            tokens before decoding. Defaults to True.

    Returns:
        text (str, list[str]): Decoded string for flat inputs or list of strings for batched inputs.
    """

    normalized = self._coerce_indices(indices)

    if isinstance(normalized, Sequence) and self._is_batch(normalized):
        return [
            self.decode(list(seq), skip_special_tokens=skip_special_tokens)  # type: ignore[list-item]
            for seq in normalized
        ]

    flat: List[int] = list(normalized)  # type: ignore[list-item]
    if skip_special_tokens:
        flat = [
            idx
            for idx in flat
            if idx not in (self.pad_idx, self.unk_idx, self.sos_idx, self.eos_idx)
        ]

    if not flat:
        return ""

    return self.sp.decode(flat)

MeCabVocab¶

MeCabVocab ¶

MeCabVocab(min_freq: Optional[int] = None, *, pad_token: Optional[str] = None, unk_token: Optional[str] = None, sos_token: Optional[str] = None, eos_token: Optional[str] = None, pad_idx: Optional[int] = None, unk_idx: Optional[int] = None, sos_idx: Optional[int] = None, eos_idx: Optional[int] = None, config: Optional[Config] = None)

Bases: BaseVocab

Vocabulary for Japanese text using MeCab morphological analysis.

Uses the fugashi library (a Python wrapper for MeCab) to tokenize Japanese text into morphemes. This is essential for Japanese NMT because Japanese does not use spaces between words.

Parameters:

Name	Type	Description	Default
`min_freq`	`int`	Minimum frequency threshold for including a token in the vocabulary. Tokens appearing fewer than min_freq times are mapped to the UNK token index. If None, uses value from config. Defaults to None.	`None`
`pad_token`	`str`	Padding token string. If None, uses value from config. Defaults to None.	`None`
`unk_token`	`str`	Unknown token string. If None, uses value from config. Defaults to None.	`None`
`sos_token`	`str`	Start-of-sequence token string. If None, uses value from config. Defaults to None.	`None`
`eos_token`	`str`	End-of-sequence token string. If None, uses value from config. Defaults to None.	`None`
`pad_idx`	`int`	Padding token index. If None, uses value from config. Defaults to None.	`None`
`unk_idx`	`int`	Unknown token index. If None, uses value from config. Defaults to None.	`None`
`sos_idx`	`int`	Start-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`eos_idx`	`int`	End-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`config`	`Config`	Configuration object to use for default values. If None, uses get_default_config(). Defaults to None.	`None`

Raises:

Type	Description
`ImportError`	If fugashi is not installed.

Attributes:

Name	Type	Description
`tagger`		The MeCab tagger instance from fugashi.
`token2idx`	`dict`	Mapping from token strings to integer indices.
`idx2token`	`dict`	Mapping from integer indices to token strings.
`token_freqs`	`dict`	Frequency count for each observed token.
`min_freq`	`int`	Minimum frequency threshold.

Examples:

>>> from torchlingo.data_processing import MeCabVocab
>>> vocab = MeCabVocab(min_freq=1)
>>> vocab.build_vocab(["私は学生です", "彼は先生です"])
>>> indices = vocab.encode("私は学生です")
>>> decoded = vocab.decode(indices)

Note

Requires installation: pip install fugashi[unidic-lite]

Source code in src/torchlingo/data_processing/vocab.py

def __init__(
    self,
    min_freq: Optional[int] = None,
    *,
    pad_token: Optional[str] = None,
    unk_token: Optional[str] = None,
    sos_token: Optional[str] = None,
    eos_token: Optional[str] = None,
    pad_idx: Optional[int] = None,
    unk_idx: Optional[int] = None,
    sos_idx: Optional[int] = None,
    eos_idx: Optional[int] = None,
    config: Optional[Config] = None,
) -> None:
    try:
        import fugashi
    except ImportError as e:
        raise ImportError(
            "MeCabVocab requires fugashi. Install with: "
            "pip install fugashi[unidic-lite]"
        ) from e

    cfg = config if config is not None else get_default_config()
    super().__init__(
        pad_token=pad_token,
        unk_token=unk_token,
        sos_token=sos_token,
        eos_token=eos_token,
        pad_idx=pad_idx,
        unk_idx=unk_idx,
        sos_idx=sos_idx,
        eos_idx=eos_idx,
        config=cfg,
    )

    self.min_freq = min_freq if min_freq is not None else cfg.min_freq
    self.tagger = fugashi.Tagger()
    self.token2idx: dict[str, int] = {
        self.pad_token: self.pad_idx,
        self.unk_token: self.unk_idx,
        self.sos_token: self.sos_idx,
        self.eos_token: self.eos_idx,
    }
    self.idx2token: dict[int, str] = {
        idx: tok for tok, idx in self.token2idx.items()
    }
    self.token_freqs: dict[str, int] = {}

build_vocab ¶

build_vocab(sentences: Sequence[str]) -> None

Build the vocabulary from a list of Japanese sentences.

Tokenizes each sentence using MeCab and counts token frequencies. Adds tokens with frequency >= min_freq to the vocabulary, skipping special tokens that are pre-initialized.

Parameters:

Name	Type	Description	Default
`sentences`	`Sequence[str]`	List of raw Japanese sentences to build vocabulary from.	required

Notes

Modifies self.token_freqs and self.token2idx in place.
Special tokens (PAD, UNK, SOS, EOS) are not overwritten.
Tokens are assigned indices in order of appearance in the iteration.

Source code in src/torchlingo/data_processing/vocab.py

def build_vocab(self, sentences: Sequence[str]) -> None:
    """Build the vocabulary from a list of Japanese sentences.

    Tokenizes each sentence using MeCab and counts token frequencies.
    Adds tokens with frequency >= min_freq to the vocabulary, skipping
    special tokens that are pre-initialized.

    Args:
        sentences (Sequence[str]): List of raw Japanese sentences to build
            vocabulary from.

    Notes:
        - Modifies self.token_freqs and self.token2idx in place.
        - Special tokens (PAD, UNK, SOS, EOS) are not overwritten.
        - Tokens are assigned indices in order of appearance in the iteration.
    """
    for sentence in sentences:
        for token in self._tokenize(sentence):
            self.token_freqs[token] = self.token_freqs.get(token, 0) + 1

    for token, freq in self.token_freqs.items():
        if freq >= self.min_freq and token not in self.token2idx:
            idx = len(self.token2idx)
            self.token2idx[token] = idx
            self.idx2token[idx] = token

len ¶

__len__() -> int

Return the total vocabulary size.

Returns:

Name	Type	Description
`int`	`int`	Number of unique tokens in the vocabulary, including special tokens.

Source code in src/torchlingo/data_processing/vocab.py

def __len__(self) -> int:
    """Return the total vocabulary size.

    Returns:
        int: Number of unique tokens in the vocabulary, including special tokens.
    """
    return len(self.token2idx)

token_to_idx ¶

token_to_idx(token: str) -> int

Convert a single token string to its vocabulary index.

Parameters:

Name	Type	Description	Default
`token`	`str`	The token to convert.	required

Returns:

Name	Type	Description
`int`	`int`	The vocabulary index of the token. Returns unk_idx if the token is not in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

def token_to_idx(self, token: str) -> int:
    """Convert a single token string to its vocabulary index.

    Args:
        token (str): The token to convert.

    Returns:
        int: The vocabulary index of the token. Returns unk_idx if the token
            is not in the vocabulary.
    """
    return self.token2idx.get(token, self.unk_idx)

idx_to_token ¶

idx_to_token(idx: int) -> str

Convert a vocabulary index to its token string.

Parameters:

Name	Type	Description	Default
`idx`	`int`	The vocabulary index.	required

Returns:

Name	Type	Description
`str`	`str`	The token string corresponding to the index. Returns unk_token if the index is not in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

def idx_to_token(self, idx: int) -> str:
    """Convert a vocabulary index to its token string.

    Args:
        idx (int): The vocabulary index.

    Returns:
        str: The token string corresponding to the index. Returns unk_token
            if the index is not in the vocabulary.
    """
    return self.idx2token.get(idx, self.unk_token)

encode ¶

encode(sentence: str, add_special_tokens: bool = True) -> List[int]

Encode a Japanese sentence to a list of vocabulary indices.

Tokenizes the sentence using MeCab and converts each token to its vocabulary index. Optionally prepends SOS and appends EOS indices.

Parameters:

Name	Type	Description	Default
`sentence`	`str`	Raw Japanese sentence string.	required
`add_special_tokens`	`bool`	If True, prepend SOS_IDX and append EOS_IDX to the sequence. Defaults to True.	`True`

Returns:

Type	Description
`List[int]`	List[int]: Encoded sequence of vocabulary indices.

Examples:

>>> vocab = MeCabVocab(min_freq=1)
>>> vocab.build_vocab(["私は学生です", "彼は先生です"])
>>> indices = vocab.encode("私は学生です", add_special_tokens=False)

Source code in src/torchlingo/data_processing/vocab.py

def encode(self, sentence: str, add_special_tokens: bool = True) -> List[int]:
    """Encode a Japanese sentence to a list of vocabulary indices.

    Tokenizes the sentence using MeCab and converts each token to its
    vocabulary index. Optionally prepends SOS and appends EOS indices.

    Args:
        sentence (str): Raw Japanese sentence string.
        add_special_tokens (bool, optional): If True, prepend SOS_IDX and
            append EOS_IDX to the sequence. Defaults to True.

    Returns:
        List[int]: Encoded sequence of vocabulary indices.

    Examples:
        >>> vocab = MeCabVocab(min_freq=1)
        >>> vocab.build_vocab(["私は学生です", "彼は先生です"])
        >>> indices = vocab.encode("私は学生です", add_special_tokens=False)
    """
    tokens = self._tokenize(sentence)
    indices = self.tokens_to_indices(tokens)
    if add_special_tokens:
        indices = [self.sos_idx] + indices + [self.eos_idx]
    return indices

decode ¶

decode(indices: IndexInput, skip_special_tokens: bool = True) -> DecodedOutput

Decode indices back to Japanese text (supports batched inputs).

Accepts a flat 1D sequence of indices or a batched 2D structure (list[list[int]] or tensor of shape [batch, seq]). Batched inputs return a list of decoded strings; flat inputs return a single string.

Note

Japanese text is reconstructed by joining tokens without spaces, which is the convention for Japanese.

Parameters:

Name	Type	Description	Default
`indices`	`IndexInput`	1D indices, 2D nested indices, or tensor.	required
`skip_special_tokens`	`bool`	If True, drop PAD/SOS/EOS/UNK tokens. Defaults to True.	`True`

Returns:

Name	Type	Description
`DecodedOutput`	`DecodedOutput`	Decoded text for each sequence.

Examples:

>>> vocab = MeCabVocab(min_freq=1)
>>> vocab.build_vocab(["私は学生です"])
>>> indices = vocab.encode("私は学生です")
>>> decoded = vocab.decode(indices)
>>> decoded
'私は学生です'

Source code in src/torchlingo/data_processing/vocab.py

def decode(
    self, indices: IndexInput, skip_special_tokens: bool = True
) -> DecodedOutput:
    """Decode indices back to Japanese text (supports batched inputs).

    Accepts a flat 1D sequence of indices or a batched 2D structure
    (list[list[int]] or tensor of shape [batch, seq]). Batched inputs
    return a list of decoded strings; flat inputs return a single string.

    Note:
        Japanese text is reconstructed by joining tokens without spaces,
        which is the convention for Japanese.

    Args:
        indices (IndexInput): 1D indices, 2D nested indices, or tensor.
        skip_special_tokens (bool, optional): If True, drop PAD/SOS/EOS/UNK
            tokens. Defaults to True.

    Returns:
        DecodedOutput: Decoded text for each sequence.

    Examples:
        >>> vocab = MeCabVocab(min_freq=1)
        >>> vocab.build_vocab(["私は学生です"])
        >>> indices = vocab.encode("私は学生です")
        >>> decoded = vocab.decode(indices)
        >>> decoded
        '私は学生です'
    """
    normalized = self._coerce_indices(indices)

    if isinstance(normalized, Sequence) and self._is_batch(normalized):
        return [
            self.decode(list(seq), skip_special_tokens=skip_special_tokens)
            for seq in normalized
        ]

    flat: List[int] = list(normalized)  # type: ignore[list-item]
    tokens = self.indices_to_tokens(flat)

    if skip_special_tokens:
        specials = {self.pad_token, self.unk_token, self.sos_token, self.eos_token}
        tokens = [tok for tok in tokens if tok not in specials]

    # Japanese text: join without spaces
    return "".join(tokens)

JiebaVocab¶

JiebaVocab ¶

JiebaVocab(min_freq: Optional[int] = None, cut_all: bool = False, use_paddle: bool = False, *, pad_token: Optional[str] = None, unk_token: Optional[str] = None, sos_token: Optional[str] = None, eos_token: Optional[str] = None, pad_idx: Optional[int] = None, unk_idx: Optional[int] = None, sos_idx: Optional[int] = None, eos_idx: Optional[int] = None, config: Optional[Config] = None)

Bases: BaseVocab

Vocabulary for Chinese text using jieba word segmentation.

Uses the jieba library to segment Chinese text into words. This is essential for Chinese NMT because Chinese does not use spaces between words.

Parameters:

Name	Type	Description	Default
`min_freq`	`int`	Minimum frequency threshold for including a token in the vocabulary. Tokens appearing fewer than min_freq times are mapped to the UNK token index. If None, uses value from config. Defaults to None.	`None`
`cut_all`	`bool`	Whether to use jieba's full mode (True) or accurate mode (False). Accurate mode is recommended for NMT. Defaults to False.	`False`
`use_paddle`	`bool`	Whether to use PaddlePaddle-based word segmentation for improved accuracy. Requires paddlepaddle. Defaults to False.	`False`
`pad_token`	`str`	Padding token string. If None, uses value from config. Defaults to None.	`None`
`unk_token`	`str`	Unknown token string. If None, uses value from config. Defaults to None.	`None`
`sos_token`	`str`	Start-of-sequence token string. If None, uses value from config. Defaults to None.	`None`
`eos_token`	`str`	End-of-sequence token string. If None, uses value from config. Defaults to None.	`None`
`pad_idx`	`int`	Padding token index. If None, uses value from config. Defaults to None.	`None`
`unk_idx`	`int`	Unknown token index. If None, uses value from config. Defaults to None.	`None`
`sos_idx`	`int`	Start-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`eos_idx`	`int`	End-of-sequence token index. If None, uses value from config. Defaults to None.	`None`
`config`	`Config`	Configuration object to use for default values. If None, uses get_default_config(). Defaults to None.	`None`

Raises:

Type	Description
`ImportError`	If jieba is not installed.

Attributes:

Name	Type	Description
`cut_all`	`bool`	Whether jieba uses full mode.
`use_paddle`	`bool`	Whether jieba uses PaddlePaddle.
`token2idx`	`dict`	Mapping from token strings to integer indices.
`idx2token`	`dict`	Mapping from integer indices to token strings.
`token_freqs`	`dict`	Frequency count for each observed token.
`min_freq`	`int`	Minimum frequency threshold.

Examples:

>>> from torchlingo.data_processing import JiebaVocab
>>> vocab = JiebaVocab(min_freq=1)
>>> vocab.build_vocab(["我是学生", "他是老师"])
>>> indices = vocab.encode("我是学生")
>>> decoded = vocab.decode(indices)

Note

Requires installation: pip install jieba

Source code in src/torchlingo/data_processing/vocab.py

def __init__(
    self,
    min_freq: Optional[int] = None,
    cut_all: bool = False,
    use_paddle: bool = False,
    *,
    pad_token: Optional[str] = None,
    unk_token: Optional[str] = None,
    sos_token: Optional[str] = None,
    eos_token: Optional[str] = None,
    pad_idx: Optional[int] = None,
    unk_idx: Optional[int] = None,
    sos_idx: Optional[int] = None,
    eos_idx: Optional[int] = None,
    config: Optional[Config] = None,
) -> None:
    try:
        import jieba as _jieba

        self._jieba = _jieba
    except ImportError as e:
        raise ImportError(
            "JiebaVocab requires jieba. Install with: pip install jieba"
        ) from e

    cfg = config if config is not None else get_default_config()
    super().__init__(
        pad_token=pad_token,
        unk_token=unk_token,
        sos_token=sos_token,
        eos_token=eos_token,
        pad_idx=pad_idx,
        unk_idx=unk_idx,
        sos_idx=sos_idx,
        eos_idx=eos_idx,
        config=cfg,
    )

    self.min_freq = min_freq if min_freq is not None else cfg.min_freq
    self.cut_all = cut_all
    self.use_paddle = use_paddle

    if use_paddle:
        self._jieba.enable_paddle()

    self.token2idx: dict[str, int] = {
        self.pad_token: self.pad_idx,
        self.unk_token: self.unk_idx,
        self.sos_token: self.sos_idx,
        self.eos_token: self.eos_idx,
    }
    self.idx2token: dict[int, str] = {
        idx: tok for tok, idx in self.token2idx.items()
    }
    self.token_freqs: dict[str, int] = {}

build_vocab ¶

build_vocab(sentences: Sequence[str]) -> None

Build the vocabulary from a list of Chinese sentences.

Tokenizes each sentence using jieba and counts token frequencies. Adds tokens with frequency >= min_freq to the vocabulary, skipping special tokens that are pre-initialized.

Parameters:

Name	Type	Description	Default
`sentences`	`Sequence[str]`	List of raw Chinese sentences to build vocabulary from.	required

Notes

Modifies self.token_freqs and self.token2idx in place.
Special tokens (PAD, UNK, SOS, EOS) are not overwritten.
Tokens are assigned indices in order of appearance in the iteration.

Source code in src/torchlingo/data_processing/vocab.py

def build_vocab(self, sentences: Sequence[str]) -> None:
    """Build the vocabulary from a list of Chinese sentences.

    Tokenizes each sentence using jieba and counts token frequencies.
    Adds tokens with frequency >= min_freq to the vocabulary, skipping
    special tokens that are pre-initialized.

    Args:
        sentences (Sequence[str]): List of raw Chinese sentences to build
            vocabulary from.

    Notes:
        - Modifies self.token_freqs and self.token2idx in place.
        - Special tokens (PAD, UNK, SOS, EOS) are not overwritten.
        - Tokens are assigned indices in order of appearance in the iteration.
    """
    for sentence in sentences:
        for token in self._tokenize(sentence):
            self.token_freqs[token] = self.token_freqs.get(token, 0) + 1

    for token, freq in self.token_freqs.items():
        if freq >= self.min_freq and token not in self.token2idx:
            idx = len(self.token2idx)
            self.token2idx[token] = idx
            self.idx2token[idx] = token

len ¶

__len__() -> int

Return the total vocabulary size.

Returns:

Name	Type	Description
`int`	`int`	Number of unique tokens in the vocabulary, including special tokens.

Source code in src/torchlingo/data_processing/vocab.py

def __len__(self) -> int:
    """Return the total vocabulary size.

    Returns:
        int: Number of unique tokens in the vocabulary, including special tokens.
    """
    return len(self.token2idx)

token_to_idx ¶

token_to_idx(token: str) -> int

Convert a single token string to its vocabulary index.

Parameters:

Name	Type	Description	Default
`token`	`str`	The token to convert.	required

Returns:

Name	Type	Description
`int`	`int`	The vocabulary index of the token. Returns unk_idx if the token is not in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

def token_to_idx(self, token: str) -> int:
    """Convert a single token string to its vocabulary index.

    Args:
        token (str): The token to convert.

    Returns:
        int: The vocabulary index of the token. Returns unk_idx if the token
            is not in the vocabulary.
    """
    return self.token2idx.get(token, self.unk_idx)

idx_to_token ¶

idx_to_token(idx: int) -> str

Convert a vocabulary index to its token string.

Parameters:

Name	Type	Description	Default
`idx`	`int`	The vocabulary index.	required

Returns:

Name	Type	Description
`str`	`str`	The token string corresponding to the index. Returns unk_token if the index is not in the vocabulary.

Source code in src/torchlingo/data_processing/vocab.py

def idx_to_token(self, idx: int) -> str:
    """Convert a vocabulary index to its token string.

    Args:
        idx (int): The vocabulary index.

    Returns:
        str: The token string corresponding to the index. Returns unk_token
            if the index is not in the vocabulary.
    """
    return self.idx2token.get(idx, self.unk_token)

encode ¶

encode(sentence: str, add_special_tokens: bool = True) -> List[int]

Encode a Chinese sentence to a list of vocabulary indices.

Tokenizes the sentence using jieba and converts each token to its vocabulary index. Optionally prepends SOS and appends EOS indices.

Parameters:

Name	Type	Description	Default
`sentence`	`str`	Raw Chinese sentence string.	required
`add_special_tokens`	`bool`	If True, prepend SOS_IDX and append EOS_IDX to the sequence. Defaults to True.	`True`

Returns:

Type	Description
`List[int]`	List[int]: Encoded sequence of vocabulary indices.

Examples:

>>> vocab = JiebaVocab(min_freq=1)
>>> vocab.build_vocab(["我是学生", "他是老师"])
>>> indices = vocab.encode("我是学生", add_special_tokens=False)

Source code in src/torchlingo/data_processing/vocab.py

def encode(self, sentence: str, add_special_tokens: bool = True) -> List[int]:
    """Encode a Chinese sentence to a list of vocabulary indices.

    Tokenizes the sentence using jieba and converts each token to its
    vocabulary index. Optionally prepends SOS and appends EOS indices.

    Args:
        sentence (str): Raw Chinese sentence string.
        add_special_tokens (bool, optional): If True, prepend SOS_IDX and
            append EOS_IDX to the sequence. Defaults to True.

    Returns:
        List[int]: Encoded sequence of vocabulary indices.

    Examples:
        >>> vocab = JiebaVocab(min_freq=1)
        >>> vocab.build_vocab(["我是学生", "他是老师"])
        >>> indices = vocab.encode("我是学生", add_special_tokens=False)
    """
    tokens = self._tokenize(sentence)
    indices = self.tokens_to_indices(tokens)
    if add_special_tokens:
        indices = [self.sos_idx] + indices + [self.eos_idx]
    return indices

decode ¶

decode(indices: IndexInput, skip_special_tokens: bool = True) -> DecodedOutput

Decode indices back to Chinese text (supports batched inputs).

Accepts a flat 1D sequence of indices or a batched 2D structure (list[list[int]] or tensor of shape [batch, seq]). Batched inputs return a list of decoded strings; flat inputs return a single string.

Note

Chinese text is reconstructed by joining tokens without spaces, which is the convention for Chinese.

Parameters:

Name	Type	Description	Default
`indices`	`IndexInput`	1D indices, 2D nested indices, or tensor.	required
`skip_special_tokens`	`bool`	If True, drop PAD/SOS/EOS/UNK tokens. Defaults to True.	`True`

Returns:

Name	Type	Description
`DecodedOutput`	`DecodedOutput`	Decoded text for each sequence.

Examples:

>>> vocab = JiebaVocab(min_freq=1)
>>> vocab.build_vocab(["我是学生"])
>>> indices = vocab.encode("我是学生")
>>> decoded = vocab.decode(indices)
>>> decoded
'我是学生'

Source code in src/torchlingo/data_processing/vocab.py

def decode(
    self, indices: IndexInput, skip_special_tokens: bool = True
) -> DecodedOutput:
    """Decode indices back to Chinese text (supports batched inputs).

    Accepts a flat 1D sequence of indices or a batched 2D structure
    (list[list[int]] or tensor of shape [batch, seq]). Batched inputs
    return a list of decoded strings; flat inputs return a single string.

    Note:
        Chinese text is reconstructed by joining tokens without spaces,
        which is the convention for Chinese.

    Args:
        indices (IndexInput): 1D indices, 2D nested indices, or tensor.
        skip_special_tokens (bool, optional): If True, drop PAD/SOS/EOS/UNK
            tokens. Defaults to True.

    Returns:
        DecodedOutput: Decoded text for each sequence.

    Examples:
        >>> vocab = JiebaVocab(min_freq=1)
        >>> vocab.build_vocab(["我是学生"])
        >>> indices = vocab.encode("我是学生")
        >>> decoded = vocab.decode(indices)
        >>> decoded
        '我是学生'
    """
    normalized = self._coerce_indices(indices)

    if isinstance(normalized, Sequence) and self._is_batch(normalized):
        return [
            self.decode(list(seq), skip_special_tokens=skip_special_tokens)
            for seq in normalized
        ]

    flat: List[int] = list(normalized)  # type: ignore[list-item]
    tokens = self.indices_to_tokens(flat)

    if skip_special_tokens:
        specials = {self.pad_token, self.unk_token, self.sos_token, self.eos_token}
        tokens = [tok for tok in tokens if tok not in specials]

    # Chinese text: join without spaces
    return "".join(tokens)

Examples¶

Building a Vocabulary¶

from torchlingo.data_processing import SimpleVocab

vocab = SimpleVocab(min_freq=2)  # Ignore words appearing < 2 times

sentences = [
    "The cat sat on the mat",
    "The dog ran in the park",
    "A bird flew over the house",
]
vocab.build_vocab(sentences)

print(f"Vocabulary size: {len(vocab)}")
print(f"Token mapping: {vocab.token2idx}")

Encoding and Decoding¶

# Encode with special tokens
indices = vocab.encode("The cat", add_special_tokens=True)
print(indices)  # [2, 4, 5, 3]  (SOS, The, cat, EOS)

# Decode back to text
text = vocab.decode(indices, skip_special_tokens=True)
print(text)  # "The cat"

Custom Special Tokens¶

vocab = SimpleVocab(
    pad_token="[PAD]",
    unk_token="[UNK]",
    sos_token="[BOS]",
    eos_token="[EOS]",
    pad_idx=0,
    unk_idx=1,
    sos_idx=2,
    eos_idx=3,
)

Using SentencePiece¶

from torchlingo.data_processing import SentencePieceVocab

# Load a pre-trained model
sp = SentencePieceVocab("data/sp_model.model")

# Tokenize (see subwords)
tokens = sp.tokenize("unfortunately")
print(tokens)  # ['▁un', 'fortun', 'ately']

# Encode to indices
indices = sp.encode("unfortunately", add_special_tokens=True)

# Decode back
text = sp.decode(indices, skip_special_tokens=True)
print(text)  # "unfortunately"

Special Tokens¶

Both vocabulary types reserve these special tokens:

Token	Default Index	Purpose
`<pad>`	0	Padding for batching
`<unk>`	1	Unknown/out-of-vocabulary words
`<sos>`	2	Start of sequence marker
`<eos>`	3	End of sequence marker

Vocabulary Comparison¶

Feature	SimpleVocab	SentencePieceVocab	MeCabVocab	JiebaVocab
Tokenization	Whitespace	BPE/Unigram subwords	Japanese morphemes	Chinese words
Unknown words	Maps to `<unk>`	Splits into subwords	Maps to `<unk>`	Maps to `<unk>`
Vocab size	Grows with data	Fixed at training	Grows with data	Grows with data
Training required	No	Yes (separate step)	No	No
Dependencies	None	`sentencepiece`	`fugashi`, `unidic-lite`	`jieba`
Best for	Small datasets, prototyping	Production, large datasets	Japanese text	Chinese text
Languages	Space-separated	Any	Japanese	Chinese, other CJK

Frequency Filtering¶

SimpleVocab supports minimum frequency filtering:

vocab = SimpleVocab(min_freq=5)  # Words must appear 5+ times
vocab.build_vocab(sentences)

# Rare words are mapped to UNK
rare_word_idx = vocab.token_to_idx("supercalifragilistic")
print(rare_word_idx == vocab.unk_idx)  # True

Saving and Loading¶

import pickle

# Save
with open("vocab.pkl", "wb") as f:
    pickle.dump(vocab, f)

# Load
with open("vocab.pkl", "rb") as f:
    vocab = pickle.load(f)

For SentencePieceVocab, the model file itself is the saved state:

# Just keep the .model file
sp = SentencePieceVocab("models/sp.model")

Asian Language Examples¶

Japanese with MeCabVocab¶

from torchlingo.data_processing import MeCabVocab

# Create and build vocabulary
vocab = MeCabVocab(min_freq=1)
sentences = [
    "私は学生です",  # "I am a student"
    "彼は先生です",  # "He is a teacher"
]
vocab.build_vocab(sentences)

# Encode (morphemes are automatically detected)
indices = vocab.encode("私は学生です", add_special_tokens=True)

# Decode (reconstructs without spaces)
text = vocab.decode(indices, skip_special_tokens=True)
print(text)  # "私は学生です"

Chinese with JiebaVocab¶

from torchlingo.data_processing import JiebaVocab

# Create and build vocabulary
vocab = JiebaVocab(min_freq=1)
sentences = [
    "我是学生",  # "I am a student"
    "他是老师",  # "He is a teacher"
]
vocab.build_vocab(sentences)

# Encode (words are automatically segmented)
indices = vocab.encode("我是学生", add_special_tokens=True)

# Decode (reconstructs without spaces)
text = vocab.decode(indices, skip_special_tokens=True)
print(text)  # "我是学生"

# Use accurate mode (default) or full mode
vocab_full = JiebaVocab(cut_all=True)  # Full segmentation mode

Installing Asian Language Support¶

# Japanese only
pip install torchlingo[japanese]

# Chinese only
pip install torchlingo[chinese]

# Both Japanese and Chinese
pip install torchlingo[asian]

Vocabulary¶

Overview¶

Quick Start¶

API Reference¶

BaseVocab¶

BaseVocab ¶

__len__ abstractmethod ¶

build_vocab abstractmethod ¶

token_to_idx abstractmethod ¶

idx_to_token abstractmethod ¶

tokens_to_indices ¶

indices_to_tokens ¶

encode abstractmethod ¶

decode abstractmethod ¶

SimpleVocab¶

SimpleVocab ¶

build_vocab ¶

__len__ ¶

token_to_idx ¶

idx_to_token ¶

encode ¶

decode ¶

SentencePieceVocab¶

SentencePieceVocab ¶

__len__ ¶

encode ¶

decode ¶

MeCabVocab¶

MeCabVocab ¶

build_vocab ¶

__len__ ¶

token_to_idx ¶

idx_to_token ¶

encode ¶

decode ¶

JiebaVocab¶

JiebaVocab ¶

build_vocab ¶

__len__ ¶

token_to_idx ¶

idx_to_token ¶

encode ¶

decode ¶

Examples¶

Building a Vocabulary¶

Encoding and Decoding¶

Custom Special Tokens¶

Using SentencePiece¶

Special Tokens¶

Vocabulary Comparison¶

Frequency Filtering¶

Saving and Loading¶

Asian Language Examples¶

Japanese with MeCabVocab¶

Chinese with JiebaVocab¶

Installing Asian Language Support¶

len `abstractmethod` ¶

build_vocab `abstractmethod` ¶

token_to_idx `abstractmethod` ¶

idx_to_token `abstractmethod` ¶

encode `abstractmethod` ¶

decode `abstractmethod` ¶

len ¶

len ¶

len ¶

len ¶