arXiv AI
Across the Loss Landscape with Progressive Growth
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 17
Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descent
arXiv:2601. 10962v2 Announce Type: replace Abstract: Stochastic gradient descent (SGD) is central to deep learning, yet the dynamical origin of its preference for flatter, more generalizable solutions remains unclear.
arXiv Machine Learning
6d ago
One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM
arXiv:2610.20415v1 Announce Type: new Abstract: Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima. Sharpness-Aware Minimisation (SAM) improves gene...
arXiv Machine Learning
Oct 9
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
The paper investigates whether flatter loss landscapes drive better generalization in the context of Grokking. It shows that sharpness‑aware minimization (SAM) alone does not reliably induce the transition to generalization, but when combined with weight decay, SAM can accelerate this transition up to four times faster. The authors provide a theoretical analysis using a minimal two‑layer ReLU model, demonstrating that flatness alone cannot distinguish memorizing from generalizing solutions, while weight decay favors generalization and a local stability window explains SAM’s effect.
arXiv Machine Learning
Jun 30
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.
arXiv AI
Jul 21
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
arXiv:2605. 29547v2 Announce Type: replace-cross Abstract: Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components such as ReLU activations and quantization operators.
arXiv Machine Learning
Jun 8
Flatland: The Adventures of Gradient Descent with Large Step Sizes
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.