arXiv Machine Learning
The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Aug 16
Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study
Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-h...
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv Machine Learning
Jun 5
Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
arXiv:2604. 00230v2 Announce Type: replace Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised.
arXiv Machine Learning
Aug 18
Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study
arXiv:2608. 15483v1 Announce Type: new Abstract: Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers.
arXiv Machine Learning
Sep 29
Common-Mode Collapse and Recovery in Direct Feedback Alignment
Direct feedback alignment (DFA) trains hidden layers via fixed random projections of output error, but with tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near a constant predictor of class frequencies. This stall is traced to the error’s common mode—a rank‑one component shared across inputs—that drives tanh units toward saturation. The study shows that calibration of the baseline readout to class priors suppresses collapse and speeds learning, while other interventions such as using Adam, adjusting feedback strength, or subtracting batch means affect the severity and recovery of collapse across MNIST, CIFAR‑10, and deeper networks.
arXiv Machine Learning
Oct 9
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
Deep neural networks can exhibit periodic loss spikes known as the "Slingshot Mechanism" during long‑term training. This paper shows that such spikes arise from finite‑precision floating‑point arithmetic: when the correct‑class logit becomes much larger than the others, the gradient for that class is rounded to zero while others remain nonzero, breaking the zero‑sum constraint and creating a positive feedback loop that inflates feature and classifier norms. The authors term this Numerical Feature Inflation (NFI) and demonstrate that it explains the rapid norm growth, reappearance of gradients, and loss spikes, while also showing that partial absorption can still cause hidden norm growth without visible spikes.