arXiv Statistics ML
A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
Hugging Face Trending Papers
Jun 3
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al. , 2017), where flatness can be measured by the trace of the Hessian of the empirical loss.
arXiv Machine Learning
Jun 4
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
arXiv:2606. 04429v1 Announce Type: cross Abstract: A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al.
arXiv Machine Learning
Jun 19
Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima
arXiv:2606. 20469v1 Announce Type: new Abstract: A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative.
arXiv AI
Jun 30
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
arXiv:2606. 28662v1 Announce Type: cross Abstract: The flatness hypothesis suggests that flatness of the loss landscape, as measured by the eigenvalues of the loss Hessian, correlates with better neural network generalization.
arXiv Machine Learning
Sep 10
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
arXiv:2609.07755v1 Announce Type: new Abstract: Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training d...
arXiv Machine Learning
Jul 16
How the Hessian-Spectrum of Neural Networks Depends on Data
arXiv:2607. 13631v1 Announce Type: new Abstract: The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc.