arXiv Machine Learning
Tracing distinguishability through transformer processing with stochastic LayerNorm
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
5d ago
Anisotropic Representation Dynamics in Transformers
The paper introduces a Mixture‑Gated Theory of Anisotropic Representation Dynamics that decomposes Transformer self‑attention into value transport, query gating, and key gating. It shows that the geometry of attended representations can cause a representation difference to contract or expand, a phenomenon termed gating‑induced regime switching. The authors derive a spectral criterion to predict switching, quantify its strength, and identify subspaces where switching occurs, validating these predictions experimentally across four Transformer architectures.
arXiv Machine Learning
Jul 2
Information-Regularized Attention for Visual-Centric Reasoning
arXiv:2607. 00434v1 Announce Type: cross Abstract: Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning.
arXiv AI
Oct 15
Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers
arXiv:2601.11618v4 Announce Type: replace-cross Abstract: An operation's input-output function does not, by itself, determine its computational role within a represented process. That role also depen...
arXiv AI
Jul 15
Sparse Inter-Layer Dependencies of Transformer FFN Neurons
arXiv:2607. 11990v1 Announce Type: cross Abstract: Feedforward network (FFN) blocks account for a large fraction of the parameters and computation in Transformer architectures, yet their internal structure remains difficult to interpret due to the additive superposition induced by the residual stream.
arXiv Computer Vision
2d ago
How Do DiTs Denoise? An Aggregator-Predictor View of Attention and MLPs
The paper investigates how diffusion transformers (DiTs) denoise images by framing them as aggregator–predictor models that use context aggregation and patch prediction. It shows that for Gaussian Markov random field priors, a global Bayes denoiser can be approximated with a limited context, and that patchwise linear denoisers recover empirical local denoisers when aggregation is informative. Experiments confirm that DiTs trained on real images exhibit noise‑dependent denoising contexts and that replacing DiT MLPs with shared‑memory patch predictors yields competitive performance, while cross‑dataset component transplantation supports the proposed functional roles of attention and MLPs.
arXiv Machine Learning
Jun 19
A Unified Perspective on the Dynamics of Deep Transformers
arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.