Position Augmentation: Reducing RoPE Extrapolation Cliffs via Random Position Scaling During Training

Transformer language models with Rotary Position Embeddings (RoPE) suffer significant performance degradation when evaluated on sequences longer than their training context window. We propose Position Augmentation (PosAug), a training-time intervention that multiplies all position indices by a random scalar $\alpha \sim U[a, b]$ at each gradient step. Unlike randomized position encodings, PosAug preserves uniform spacing between adjacent tokens while exposing the model to a range of effective RoPE frequency scales. At Chinchilla-ratio training budgets ($\sim$20 tokens/parameter) with $n=3$ seeds, PosAug reduces the extrapolation cliff by $43\times$ at 42M parameters (cliff $2.65 \to 0.06$) and $14\times$ at 113M ($2.84 \to 0.20$), with $\leq 1.4\%$ in-distribution penalty and $< 0.5\%$ wall-clock overhead. The method composes with inference-time scaling (PosAug+YaRN) and with longer context windows. The positional range exposed during training follows the approximate heuristic $L_{\max} \approx b \times w$.