arXiv Machine Learning
Sharpness-aware Orthogonalized Momentum Algorithms
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 9
Convergence Bound and Critical Batch Size of Muon Optimizer
arXiv:2507. 01598v5 Announce Type: replace Abstract: Muon, a recently proposed optimizer that leverages the inherent matrix structure of neural network parameters, has demonstrated strong empirical performance, indicating its potential as a successor to standard optimizers such as AdamW.
arXiv Machine Learning
Jul 24
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
arXiv:2605. 11396v2 Announce Type: replace Abstract: The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization.
arXiv Machine Learning
Jul 8
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
arXiv Machine Learning
6d ago
When and How Does Muon Precondition? The Role of Structured Gradient Noise
arXiv:2610.31728v1 Announce Type: new Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study...
arXiv Machine Learning
Aug 18
EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations
arXiv:2608. 15105v1 Announce Type: new Abstract: Recent progress in optimization research has highlighted the sharpness of the loss landscape as a key factor in narrowing the generalization gap.
arXiv AI
Jun 2
Stability Analysis of Sharpness-Aware Minimization
arXiv:2301. 06308v2 Announce Type: replace-cross Abstract: Sharpness-aware minimization (SAM) is a training method that seeks to find flat minima in deep learning, resulting in state-of-the-art performance across various domains.