A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone

Transformers are built with continuous symmetries in the attention mechanism. We derive ρ, a scalar invariant that measures each head's capacity to couple to rotary positional encodings. This framework allows us to analyze heads without forward passes, opening a new research direction into data-independent analysis of modern LLMs. Empirically, we find that ρ predicts induction heads, the most-studied circuit type in mechanistic interpretability, with Spearman correlation between -0.55 and -0.74 across four bias-free RoPE models spanning 135M to 7B parameters. As a preliminary proof of concept, we use ρ to apply YaRN to only the top-50% positional heads, which degrades perplexity less than uniform application. Code is available at https://github.com/MPedraBento/rho-invariant.