arXiv Machine Learning July 24, 2026 By Xiaolong Li, Zhangchen Zhou, Zhi-Qin John Xu

Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

Read the original on arXiv Machine Learning →

arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 30

Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.

By Etienne Boursier, Matthew Bowditch, Matthias Englert, Ranko Lazic

arXiv Machine Learning

Jun 5

Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime

arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.

By Leonardo Defilippis, Yizhou Xu, Julius Girardin, Emanuele Troiani, Vittorio Erba, Lenka Zdeborov\'a, Bruno Loureiro, Florent Krzakala

arXiv AI

Jul 28

Scale Weight Decay and Train Better

arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.

By Anuj Apte

safety

arXiv Machine Learning

Jul 21

Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks

arXiv:2607. 16720v1 Announce Type: new Abstract: Understanding deep neural networks remains a central challenge in machine learning.

By Haruka Eshima, Makoto Yamada

arXiv Machine Learning

Jul 9

Network Dynamics-Based Framework for Understanding Deep Neural Networks

arXiv:2501. 02436v5 Announce Type: replace Abstract: Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning.

By Yuchen Lin, Yong Zhang, Sihan Feng, Hong Zhao

fine-tuning

arXiv Machine Learning

Jun 5

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

arXiv:2604. 00230v2 Announce Type: replace Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised.

By Anamika Paul Rupa

Related stories

Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime

Scale Weight Decay and Train Better

Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks

Network Dynamics-Based Framework for Understanding Deep Neural Networks

Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold