arXiv Machine Learning
Calibrated Target Noise Recovers Curvature from the Gradients
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 2
Exploiting weight-space symmetries for approximating curvature
arXiv:2606. 00442v1 Announce Type: new Abstract: Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of modern deep networks.
Hugging Face Trending Papers
Jun 29
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there. We introduce Curvature-Weighted Gradient Diversity (CWGD), a geometry-aware measure that weights per-sample gradient diversity by the inverse square root of the Hessian, providing a tighter proxy for the effective optimization noise.
arXiv Machine Learning
Jun 30
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
arXiv:2606. 30455v1 Announce Type: new Abstract: The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there.
arXiv Machine Learning
Aug 26
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...
arXiv Machine Learning
Jun 26
Stochastic Gradient Optimization with Model-Assisted Sampling
arXiv:2606. 27171v1 Announce Type: new Abstract: This work addresses the problem of variance in stochastic gradient estimation for machine learning optimization.
arXiv Statistics ML
Oct 2
Exact information accounting for SGD methods
The paper presents an exact, information‑theoretic analysis of stochastic gradient descent (SGD) and its variants, showing that a preconditioned SGD step corresponds to a posterior‑mean update in a Gaussian Bayes model. It decomposes one‑step regret into an intrinsic‑time cost and a change in comparator information, extending this split to an identity for the objective itself. The framework links convex convergence, saddle‑point escape, flatness‑generalization trade‑offs, learning‑rate schedules, adaptive optimizers, and various SGD variants, and it is validated on synthetic and real training runs, revealing how different optimizers achieve the same training loss through distinct step characteristics.