Positional Encoding¶

Position encoding implementation for Transformer models.

Overview¶

Transformers process all positions in parallel and have no inherent sense of order. Positional encodings add position information to token embeddings so the model can distinguish between "The cat sat" and "sat cat The".

TorchLingo implements the sinusoidal positional encoding from the original Transformer paper ("Attention Is All You Need", Vaswani et al., 2017).

Quick Start¶

from torchlingo.models.positional import SinusoidalPositionalEncoding
import torch

pos_enc = SinusoidalPositionalEncoding(d_model=512, max_seq_len=2048)

# Apply to embeddings
embeddings = torch.randn(2, 100, 512)  # [batch, seq_len, d_model]
positioned = pos_enc(embeddings)

API Reference¶

SinusoidalPositionalEncoding ¶

SinusoidalPositionalEncoding(d_model: Optional[int] = None, max_seq_len: Optional[int] = None, dropout: Optional[float] = None, base: float = 10000.0, config: Optional[Config] = None)

Bases: Module

Fixed sinusoidal positional encoding added to token embeddings.

For position pos and dimension index i, the encoding is:

PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

The encodings are precomputed once for max_seq_len positions and stored in a buffer; the forward pass simply slices and adds them to the input embeddings, followed by dropout. If a longer sequence is seen at runtime, the table is rebuilt on the input's device.

Parameters:

Name	Type	Description	Default
`d_model`	`int`	Embedding dimension. Falls back to config.d_model.	`None`
`max_seq_len`	`int`	Number of positions to precompute. Falls back to config.max_seq_length.	`None`
`dropout`	`float`	Dropout applied after adding the encoding. Falls back to config.dropout.	`None`
`base`	`float`	Base for the geometric progression of wavelengths. Defaults to 10000.0.	`10000.0`
`config`	`Config`	Configuration object. Defaults to default config.	`None`

Attributes:

Name	Type	Description
`d_model`	`int`	Embedding dimension.
`max_seq_len`	`int`	Number of precomputed positions.
`base`	`float`	Wavelength base.
`pe`	`Tensor`	Precomputed encodings of shape (max_seq_len, d_model), registered as a non-persistent buffer.

Source code in src/torchlingo/models/positional.py

def __init__(
    self,
    d_model: Optional[int] = None,
    max_seq_len: Optional[int] = None,
    dropout: Optional[float] = None,
    base: float = 10000.0,
    config: Optional[Config] = None,
) -> None:
    super().__init__()
    cfg = config if config is not None else get_default_config()
    d_model = d_model if d_model is not None else cfg.d_model
    max_seq_len = max_seq_len if max_seq_len is not None else cfg.max_seq_length
    dropout = dropout if dropout is not None else cfg.dropout

    self.d_model = d_model
    self.max_seq_len = max_seq_len
    self.base = base
    self.dropout = nn.Dropout(p=dropout)
    self.register_buffer(
        "pe", self._build_table(max_seq_len, d_model, base), persistent=False
    )

forward ¶

forward(x: Tensor) -> Tensor

Add positional encodings to a batch of embeddings.

Parameters:

Name	Type	Description	Default
`x`	`Tensor`	Token embeddings of shape (batch_size, seq_len, d_model).	required

Returns:

Type	Description
`Tensor`	torch.Tensor: Embeddings with positional information added, same shape as the input, after dropout.

Source code in src/torchlingo/models/positional.py

def forward(self, x: torch.Tensor) -> torch.Tensor:
    """Add positional encodings to a batch of embeddings.

    Args:
        x (torch.Tensor): Token embeddings of shape
            (batch_size, seq_len, d_model).

    Returns:
        torch.Tensor: Embeddings with positional information added,
            same shape as the input, after dropout.
    """
    seq_len = x.size(1)
    if seq_len > self.pe.size(0):
        # Rebuild on the input's device so the cache never lags behind
        # a model.to(device) move.
        self.pe = self._build_table(seq_len, self.d_model, self.base, x.device)
    return self.dropout(x + self.pe[:seq_len].unsqueeze(0))

How Positional Encoding Works¶

The Problem¶

Attention computes:

\[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]

This is permutation invariant—shuffling the input gives shuffled output with the same attention weights. We need to inject position information.

The Sinusoidal Solution¶

The original Transformer uses fixed sinusoidal patterns. For position \(pos\) and dimension pair \(i\):

\[ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right) \]

\[ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right) \]

Each dimension pair oscillates at a different wavelength, from \(2\pi\) up to \(10000 \cdot 2\pi\), so every position gets a unique fingerprint. The encoding is simply added to the token embeddings:

embeddings = token_embeddings + position_encodings

Key insight: for any fixed offset \(k\), \(PE_{pos+k}\) is a linear function of \(PE_{pos}\) (a rotation in each sine/cosine pair). This makes it easy for the model to learn to attend by relative position — which is what matters for translation ("the adjective comes right before the noun").

Alternatives You'll See in the Wild¶

Method	Idea	Parameters
Sinusoidal (this library)	Fixed sine/cosine waves added to embeddings	None
Learned	`nn.Embedding(max_len, d_model)` added to embeddings	max_len × d_model
Rotary (RoPE)	Rotate the query/key vectors inside attention so dot products depend only on relative offsets	None

Learned encodings can't represent positions beyond the trained maximum. RoPE is popular in modern LLMs but requires a custom attention implementation (it must be applied to Q and K inside every attention layer, not to the input embeddings) — a great extension exercise once you understand the sinusoidal version.

Usage in SimpleTransformer¶

SimpleTransformer applies the encoding automatically:

model = SimpleTransformer(
    src_vocab_size=10000,
    tgt_vocab_size=10000,
    d_model=512,
    max_seq_length=2048,  # positions precomputed up to this length
)

Token embeddings are scaled by \(\sqrt{d_{model}}\) (so they don't drown out the position signal) and the encoding is added in the _embed method before the encoder/decoder layers. Dropout is applied to the sum, as in the original paper.

Handling Long Sequences¶

The table is precomputed for max_seq_len positions but extends itself automatically if a longer input arrives:

pos_enc = SinusoidalPositionalEncoding(d_model=512, max_seq_len=512)
long_seq = torch.randn(1, 1500, 512)  # Beyond 512
positioned = pos_enc(long_seq)  # Table is rebuilt to 1500 positions

Because the encoding is a fixed function of position, it produces valid values for any position. Model quality at positions far beyond those seen in training is a separate question — for best results, train with sequences close to your expected inference length.