The paper presents an exact, information‑theoretic analysis of stochastic gradient descent (SGD) and its variants, showing that a preconditioned SGD step corresponds to a posterior‑mean update in a Gaussian Bayes model. It decomposes one‑step regret into an intrinsic‑time cost and a change in comparator information, extending this split to an identity for the objective itself. The framework links convex convergence, saddle‑point escape, flatness‑generalization trade‑offs, learning‑rate schedules, adaptive optimizers, and various SGD variants, and it is validated on synthetic and real training runs, revealing how different optimizers achieve the same training loss through distinct step characteristics.
By Akshay Balsubramani
arXiv:2602. 05600v2 Announce Type: replace Abstract: Stochastic Gradient Descent (SGD) introduces anisotropic noise that is correlated with the local curvature of the loss landscape, thereby biasing optimization toward flat minima.
By Yikuan Zhang, Ning Yang, Yuhai Tu
arXiv:2606. 18306v1 Announce Type: new Abstract: Gaussian width is a central geometric complexity measure in high-dimensional probability, compressed sensing, convex optimization, and learning theory.
By Vu Khac Ky
arXiv:2510. 07758v3 Announce Type: replace Abstract: Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks.
By Qiaozhe Zhang, Jun Sun, Ruijie Zhang, Yingzhuang Liu
arXiv:2609.31101v1 Announce Type: cross
Abstract: The flatness of the loss landscape at a minimizer is a widely used heuristic for reasoning about neural-network generalization, yet evidence for this...
By Brandon Livio Annesi, Davide Straziota, Enrico Maria Malatesta
arXiv:2607. 08380v1 Announce Type: new Abstract: An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian.
By Lachlan Ewen MacDonald, Ren\'e Vidal
The paper investigates how stochastic gradient descent (SGD) selects specific functional decompositions when training a deep linear residual network to learn the identity function. Although many weight configurations minimize the population loss, SGD consistently prefers particular solutions, especially under anisotropic label noise or different parametrizations. The authors explain this bias using an entropic loss term that penalizes the expected squared norm of the minibatch gradient, analytically characterizing its minimizers and showing that trained networks align with these predictions.
By Andy Arditi, Weian Xie, David Bau, Liu Ziyin
arXiv:2604. 03146v2 Announce Type: replace-cross Abstract: We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs.
By Chiheb Yaakoubi, Cosme Louart, Malik Tiomoko, Zhenyu Liao
arXiv:2606. 06772v2 Announce Type: replace-cross Abstract: Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning.
By Junyu Zhou, Puyu Wang, Dennis Wagner, Yunwen Lei, Marius Kloft, Yiming Ying
arXiv:2607. 20578v1 Announce Type: new Abstract: We study Gaussian-width complexity on statistical manifolds through a pair of functionals: the primal Fisher width $w_G(T) = w(G^{1/2}T)$, induced by the Fisher metric, and the inverse-Fisher width $w_{G^{-1}}(T) = w(G^{-1/2}T)$, induced by the inverse Fisher metric.
By Vu Khac Ky
arXiv:2606. 06772v1 Announce Type: cross Abstract: Understanding the generalization performance of over-parameterized neural networks has become a central topic in deep learning theory.
By Junyu Zhou, Puyu Wang, Yunwen Lei, Marius Kloft, Yiming Ying
arXiv:2606. 19105v1 Announce Type: new Abstract: We study PAC-Bayes derandomization for smooth loss functions.
By Alexandre Lemire Paquin, Brahim Chaib-Draa, Philippe Gigu\`ere