arXiv Machine Learning
Hybrid Momentum Methods for Optimizing Deep Learning Models
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 25
A Physical Response-and-Memory Model for Muon Optimization
The paper introduces a physical response-and-memory model for the Muon optimizer, explaining its semi‑orthogonalized momentum update as the maximally dissipative direction under an output‑side safety budget. It treats the weight matrix as a responsive medium with internal stress, showing that momentum corresponds to accumulated stress whose relaxation occurs over multiple timescales—fast and slow. Based on this, the authors propose the Bi‑Maxwell optimizer, which uses a two‑timescale memory kernel and achieves target loss in fewer steps on a public large‑language‑model benchmark.
arXiv Machine Learning
Jun 17
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
arXiv Machine Learning
Aug 4
AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.
arXiv AI
Jun 24
Fast and Slow Variational Continual Learning
arXiv:2606. 24007v1 Announce Type: cross Abstract: Continual learning remains a major challenge for modern deep networks, partly because commonly used optimizers lack inherent mechanisms for continual adaptation.
arXiv Machine Learning
Aug 18
Adaptive Optimization via Momentum on Variance-Normalized Gradients
arXiv:2602. 10204v2 Announce Type: replace Abstract: We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization.
arXiv Machine Learning
6d ago
When and How Does Muon Precondition? The Role of Structured Gradient Noise
arXiv:2610.31728v1 Announce Type: new Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study...