Uncovering hidden geometry in Transformers via disentangling position and context

Transformers are widely used to extract semantic meanings from input tokens, yet they usually operate as black-box models. In this paper, we present a simple yet informative decomposition of hidden states (or embeddings) of trained transformers into interpretable components. For any layer, embedding vectors of input sequence samples are represented by a tensor h \in \mathbb{R}^{C \times T \times d}. Given embedding vector h_{c,t} \in \mathbb{R}^d at sequence position t \le T in a sequence (or context) c \le C, extracting the mean effects yields the decomposition [ h_{c,t} = \mu + pos_t + ctx_c + resid_{c,t} ] where \mu is the global mean vector, pos_t and ctx_c are the mean vectors across contexts and across positions respectively, and resid_{c,t} is the residual vector. For popular transformer architectures and diverse text datasets, empirically we find pervasive mathematical structure: (1) (pos_t)_{t} forms a low-dimensional, continuous, and often spiral shape across layers, (2) (ctx_c)c shows clear cluster structure that falls into context topics, and (3) (pos_t){t} and (ctx_c)_c are mutually nearly orthogonal. We argue that smoothness is pervasive and beneficial to transformers trained on languages, and our decomposition leads to improved model interpretability.