arXiv Statistics ML
Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Sep 7
Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
The paper introduces Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training that turns hierarchical feature learning into an explicit iterative spectral procedure. In this framework, each layer independently selects directions with maximal low-degree correlation to the label, providing a tractable surrogate for deep learning and a kernel-space interpretation. Experiments on fully connected and convolutional networks show that Neural LoFi outperforms lazy random-feature baselines, recovers meaningful structured filters, and aligns with early gradient-descent feature discovery on real datasets.
arXiv Statistics ML
Aug 25
A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks
The paper introduces a finite‑width geometric framework that explains how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. It quantifies incompatibilities among weight‑generated covariance, gates, and backward sensitivities using three families of commutators, and provides exact layerwise identities that decompose these commutators into sources such as downstream transport, adjacent‑layer imbalance, and nonlinear gate‑covariance interactions. The study demonstrates that spectral alignment is a layer‑ and scale‑dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping, rather than a universal consequence of training.
Hugging Face Trending Papers
Aug 24
A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks
We develop a finite-width geometric framework describing how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. Incompatibility among weight-genera...
arXiv Statistics ML
Oct 26
Singular Geometry of Low-Rank Adaptation
The paper investigates the singular statistical nature of Low‑Rank Adaptation (LoRA), a popular parameter‑efficient fine‑tuning method for large neural networks. By applying the real log canonical threshold (RLCT) from singular learning theory, the authors analyze how LoRA’s learning complexity depends on its low‑rank factorization and the pretrained network. They show that the forward‑backward covariance, derived from activations and backpropagated sensitivities, governs which low‑rank perturbations are statistically observable, leading to a detailed characterization of the model’s singular geometry both locally and globally, and introduce a curvature‑corrected estimator for the RLCT.
arXiv Machine Learning
Jun 10
Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks
arXiv:2606. 10324v1 Announce Type: new Abstract: The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.