arXiv Machine Learning
The Geometry of Inference in Transformer Residual Streams
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 22
An Analysis of Residual-Stream Geometry Across Transformer Depth
arXiv:2607. 18348v1 Announce Type: cross Abstract: We propose a transition-centred geometric analysis of transformer residual streams.
arXiv Machine Learning
Aug 14
Geometric and Behavioral Stratification in Transformer Residual Streams
arXiv:2608. 12447v1 Announce Type: new Abstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream.
arXiv Machine Learning
2d ago
The Residual Stream's Effective Depth
The paper introduces effective depth (Deff), a scalar diagnostic that treats a transformer’s layer‑wise residual stream as a discrete‑time process and measures how representation similarity decays with layer distance. Across sixteen decoder‑only language models, Deff reveals that most models exhibit a lower similarity decay than the closed‑form reference, indicating correlated residual updates rather than unused depth. The study also shows that this effect is robust to various controls and persists early in training, suggesting Deff is a global accumulated‑state diagnostic rather than a capability score.
arXiv AI
Jul 21
Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis
arXiv:2606. 07559v2 Announce Type: replace-cross Abstract: Fine-tuning a language model often fails silently when its correct completion must outrank a near-synonym competitor.
arXiv AI
Sep 15
Ventriloquist LLMs: Linear Alignment of Late-Stage Representations
arXiv:2603.18908v5 Announce Type: replace Abstract: Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures,...
arXiv AI
Jun 9
Phantom transitions in language model fine-tuning
arXiv:2606. 07559v1 Announce Type: cross Abstract: Fine-tuning a language model on contexts whose correct completion has a near-synonym competitor often fails silently.