arXiv Machine Learning July 2, 2026 By Minhak Song, Liang Zhang, Bingcong Li, Niao He, Michael Muehlebach, Sewoong Oh

Zeroth-Order Optimization at the Edge of Stability

Read the original on arXiv Machine Learning →

arXiv:2604. 14669v2 Announce Type: replace Abstract: Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

fine-tuning

arXiv Machine Learning

Jun 8

Flatland: The Adventures of Gradient Descent with Large Step Sizes

arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.

By Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus, Mark Schmidt, Holger Rauhut

arXiv Machine Learning

Jun 11

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.

By Feihu Jin, Shipeng Cen, Ying Tan

llms fine-tuning benchmarks

arXiv Machine Learning

6d ago

A New First-Order Meta-Learning Algorithm with Convergence Guarantees

arXiv:2409. 03682v2 Announce Type: replace Abstract: Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems.

By El Mahdi Chayti, Martin Jaggi

llms efficiency safety

arXiv AI

Jun 3

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

arXiv:2606. 02857v1 Announce Type: cross Abstract: Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation.

By Liyan Tan, Yequan Zhao, Yifan Yang, Ruijie Zhang, Xinling Yu, Zheng Zhang

llms fine-tuning efficiency

arXiv Machine Learning

Aug 5

On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

arXiv:2608. 03197v1 Announce Type: new Abstract: Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear.

By Jiaxin Deng, Junbiao Pang

safety

arXiv Machine Learning

1d ago

On the Principles Behind Neural Network Optimizers

arXiv:2608. 16760v1 Announce Type: new Abstract: Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation.

By Yushun Zhang

llms efficiency

Related stories

Flatland: The Adventures of Gradient Descent with Large Step Sizes

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

A New First-Order Meta-Learning Algorithm with Convergence Guarantees

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

On the Principles Behind Neural Network Optimizers