The Flow has not summarised this story yet — read it at arXiv Statistics ML.
arXiv:2601. 22652v2 Announce Type: replace-cross Abstract: Spectral gradient methods, such as the Muon optimizer, modify gradient updates by preserving directional information while discarding scale, and have shown strong empirical performance in deep learning.
arXiv:2501. 02436v5 Announce Type: replace Abstract: Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning.
arXiv:2401. 04013v2 Announce Type: replace Abstract: Deep learning models, such as wide neural networks, can be conceptualized as nonlinear dynamical physical systems characterized by a multitude of interacting degrees of freedom.
arXiv:2607. 12332v1 Announce Type: new Abstract: We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization.
We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization. Extending Theorem 1 from Pesme & Flammarion (2023), we generalize the analysis to both deep diagonal linear networks and a broader class of two-layer diagonal linear networks (as defined in Definition 4.
The paper analyzes gradient flow in a two‑layer linear neural network as the input, hidden, and output dimensions grow proportionally. Using a closed‑form solution combined with random matrix theory, it derives exact asymptotic spectral distributions for the weight matrices at all times and tracks how the learned subspace evolves. In the limit of small initialization, the study reveals a silent alignment phase where singular directions align quickly while singular values and loss stay unchanged, followed by a visible learning phase on a logarithmic timescale where singular values increase and loss decreases through a saddle‑to‑saddle dynamic that becomes continuous in the high‑dimensional limit.