arXiv Machine Learning August 5, 2026 By Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi

Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability

Read the original on arXiv Machine Learning →

arXiv:2511. 19716v3 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

benchmarks

arXiv Machine Learning

Jul 15

Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

arXiv:2607. 12360v1 Announce Type: new Abstract: The cooldown phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining, lowers the final training loss in some settings and does nothing in others.

By Subham Singh, Ashutosh Mishra, Subha Raut

arXiv Machine Learning

Jul 28

Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity

arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.

By Arzu Ahmadova, Ismail Huseynov

benchmarks

Hugging Face Trending Papers

Jul 7

On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs. However, in ill-conditioned or non-convex landscapes, the implicit inverse Hessian approximation can suffer from an exploding condition number, leading to numerical instability and degraded convergence.

benchmarks

arXiv Machine Learning

5d ago

When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization

arXiv:2609. 30271v1 Announce Type: new Abstract: Adaptive optimizers are commonly parameterized by a fixed power of the second-moment estimate.

By Gongyue Zhang, Honghai Liu

benchmarks

arXiv Machine Learning

4d ago

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

arXiv:2505.20817v3 Announce Type: replace-cross Abstract: Gradient clipping is widely used in language-model training to control heavy-tailed gradient noise and can improve convergence guarantees ove...

By Taha El Bakkali El Kadi, Savelii Chezhegov, Aleksandr Beznosikov, Samuel Horv\'ath, Eduard Gorbunov

safety

arXiv Machine Learning

Jul 20

Regularity-Aware Stochastic MGDA with Adaptive Conflict-Avoidant Update Direction Control

arXiv:2607. 15412v1 Announce Type: new Abstract: Multi-objective learning (MOL) aims to optimize multiple objectives simultaneously.

By Chentong Huang, Lisha Chen

safety

One email a morning, machine-written

Related stories

Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity

On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization

High-Probability Convergence of Clipped SGD under Heavy-Tailed Noise and $(L_0,L_1)$-Smoothness

Regularity-Aware Stochastic MGDA with Adaptive Conflict-Avoidant Update Direction Control