Hugging Face Trending Papers

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points. No existing work has defined a measurable RG order parameter, tested it under controlled variation of the input distribution, or made quantitative predictions that are empirically verified.

arXiv Machine Learning

Jun 10

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

arXiv:2606. 10324v1 Announce Type: new Abstract: The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points.

By Parviz Haggi-Mani, Irina Rish

arXiv AI

Jul 16

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.

By Katie Everett

llms

Hugging Face Trending Papers

Jul 15

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.

llms

arXiv Machine Learning

Jun 10

Overcoming Rank Collapse in Feedback Alignment

arXiv:2606. 11123v1 Announce Type: new Abstract: Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation.

By Gauthier Boeshertz, Razvan Pascanu, Claudia Clopath

benchmarks safety

arXiv AI

Jun 16

Scaling Adaptive Depth with Norm-Agnostic Residual Networks

arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.

By Tom\'as Figliolia, Beren Millidge

llms efficiency

arXiv Machine Learning

1d ago

Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

arXiv:2608. 14664v1 Announce Type: new Abstract: How can we determine whether a trained neural network is already deep enough?

By Zeyu Liu, Jinhao Zhang, Yunquan Zhang, Guangming Tan, Xiang Gao, Fangming Liu, Daning Cheng

llms

arXiv AI

Jun 26

CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.

By Huzama Ahmad, Cao Viet Hai Nam, Se-Young Yun

llms efficiency

arXiv Machine Learning

Jul 15

Fisher Rank Inflation: A Spectral Signature of Memorization under Label Noise

arXiv:2607. 12438v1 Announce Type: new Abstract: Deep networks trained with label noise often learn clean structure before memorizing corrupted labels.

By Satwik Bathula, Anand A. Joshi

llms computer-vision

arXiv Machine Learning

Jul 20

Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention

arXiv:2607. 15449v1 Announce Type: new Abstract: Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator.

By Parviz Haggi-Mani, Irina Rish

llms

arXiv Machine Learning

5d ago

Geometric and Behavioral Stratification in Transformer Residual Streams

arXiv:2608. 12447v1 Announce Type: new Abstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream.

By Nelson Guda

llms

arXiv Machine Learning

Jun 29

Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks

arXiv:2606. 27759v1 Announce Type: new Abstract: Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE), whose forward/backward mismatch produces severe accuracy degradation as networks deepen.

By Evan Gibson Smith, Bashima Islam

fine-tuning

arXiv Machine Learning

Jun 5

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.

By Hu Tan, Kuo Gai, Shihua Zhang

rag

Related stories