Mechanistic interpretability has identified transformer circuits, but lacks a shared vocabulary for describing how their functions compose across tasks and architectures. We introduce Coherentist Prob...
arXiv:2608. 12447v1 Announce Type: new Abstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream.
The paper introduces Conditional Functional Substitutability (CFS) as a new way to measure redundancy in Transformers by examining when intermediate states produce similar downstream responses. CFS uncovers functional relationships and potential reductions that traditional importance- or similarity-based metrics miss, revealing systematic reorganization as models scale. Experiments across modalities and Transformer families show that performance gains do not always align with increased substitutability, and that models with more independent functional structure perform better, offering a functional explanation for diminishing returns and enabling more efficient dynamic computation.
arXiv:2608. 01968v1 Announce Type: new Abstract: Transformer models are most often understood through what they do: their benchmark performance, generation quality, or behavior on downstream tasks.
arXiv:2609.16537v1 Announce Type: cross Abstract: Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical kn...
arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
arXiv:2605.01609v2 Announce Type: replace-cross Abstract: We find that transformer concept representations systematically anti-concentrate in the spectral tail of the unembedding covariance, encoding...
arXiv:2609.15975v1 Announce Type: cross Abstract: Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study thi...
arXiv:2603. 13259v2 Announce Type: replace-cross Abstract: When a decoder-only transformer is forced to process matched correct and incorrect single-token continuations of a factual query, the two pathways through hidden-state space diverge in a specific way: displacement vectors from the query-only representation maintain approximately equal magnitude but rotate apart in direction.
The paper introduces the Communication Map, a method that charts every potential communication channel in a transformer model using only its weights. It generalizes previous coupling metrics into a single coefficient covering all 18 connection classes, revealing that 70‑89% of head pairs are non‑randomly oriented and identifying strong or avoiding couplings. The authors demonstrate the map’s utility by recovering known induction circuits and uncovering a two‑dimensional stream subspace whose removal eliminates induction capabilities across several models.
arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...