0

Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable

Möbius RoPE is a rotary positional encoding built on the anti-periodic frequency ladder $θ_i=π(2i+1)/N$: every rotation plane advances by an odd multiple of $π$ across the training context, so the positional holonomy is $-1$ and the two ends of the sequence are deterministically…

Preview
Year
2026
Hosting
Full text hostedCC-BY-4.0

Cite

Notes

Only stored in your browser.

Attribution

Abstract & full text
arxiv.org/abs/2607.21405CC-BY-4.0
TL;DR
Semantic Scholar
Attribution policy →

Abstract

Möbius RoPE is a rotary positional encoding built on the anti-periodic frequency ladder θ_i=π(2i+1)/N: every rotation plane advances by an odd multiple of π across the training context, so the positional holonomy is -1 and the two ends of the sequence are deterministically coupled through a closed-form Dirichlet "dipole"; to our knowledge this is the first anti-periodic boundary condition in positional encoding. We verify the theory numerically to \sim 10^{-6} and pretrain 48 models spanning six 160M-class and three 410M-class arms (2B FineWeb-Edu tokens each; the hybrid arm puts Möbius frequencies on 25% of heads). Hybrid perplexity is unchanged (29.66 vs. 29.72), but needle-in-a-haystack retrieval becomes reliable: 90.3\pm5.7% versus 63.3\pm31.4% at context 512 (n=6 seeds), observed worst seed 86% versus 14%, robust variance tests p=0.013-0.029 (unadjusted), recurring at 410M (Levene p=0.040). Matched controls isolate the mechanism: an aperiodic ladder in the same frequency band reproduces none of the effect, and a periodic (holonomy +1) ladder only a fraction. Swapping trained models' frequency table back to standard RoPE (weights frozen) collapses retrieval, with damage concentrated on far needles: trained models depend on this long-range geometry. A NoPE arm is even more reliable at short context but pays a 13% perplexity tax and extrapolates worst; only the anti-periodic hybrid pairs baseline perplexity with a high reliability floor. The effect is scoped to single-needle retrieval within the training window; a one-line frequency swap thus provides zero-cost insurance against the retrieval seed lottery.