arXiv Machine Learning

The Geometry of Inference in Transformer Residual Streams

arXiv Machine Learning
Sep 29

The Residual Stream's Effective Depth

The paper introduces effective depth (Deff), a scalar diagnostic that treats a transformer’s layer‑wise residual stream as a discrete‑time process and measures how representation similarity decays with layer distance. Across sixteen decoder‑only language models, Deff reveals that most models exhibit a lower similarity decay than the closed‑form reference, indicating correlated residual updates rather than unused depth. The study also shows that this effect is robust to various controls and persists early in training, suggesting Deff is a global accumulated‑state diagnostic rather than a capability score.

By Barak Gahtan, Ido Galil, Alex M. Bronstein
arXiv AI
Oct 27

Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones

The paper presents quantitative generalization bounds for fixed‑precision transformer models, addressing both in‑distribution statistical generalization and length generalization. It derives guarantees based on model parameters, quantization space, and alphabet size, and shows how training length and sample size trade off, with longer training being exponentially more efficient. Experiments confirm the theoretical predictions.

By Marius Henry, Michael Rizvi-Martel, Michael Hahn