arXiv Machine Learning July 2, 2026 By Xun Dong, Yibo Xu, Naigang Wang, Xin Li, Penghang Yin, Zi Yang

ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

Read the original on arXiv Machine Learning →

arXiv:2607. 01125v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 5

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.

By Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin, Deliang Fan, Li Yang

llms fine-tuning benchmarks

arXiv Machine Learning

1d ago

SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates

arXiv:2608. 15665v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates.

By Ziming Yu, Shuyao Xiao, Xingyu Zhao, Sike Wang, Pan Zhou, Peiyu Zang, Xiangda Yan, Yongjie Yang, Jia Li

llms fine-tuning

arXiv Machine Learning

Jun 11

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.

By Feihu Jin, Shipeng Cen, Ying Tan

llms fine-tuning benchmarks

arXiv AI

Jun 3

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

arXiv:2606. 02857v1 Announce Type: cross Abstract: Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation.

By Liyan Tan, Yequan Zhao, Yifan Yang, Ruijie Zhang, Xinling Yu, Zheng Zhang

llms fine-tuning efficiency

arXiv Machine Learning

Jun 16

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.

By Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin, Aleksandr Beznosikov

llms fine-tuning

arXiv AI

Jun 10

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning

arXiv:2606. 10989v1 Announce Type: new Abstract: Large language model unlearning aims to suppress designated undesirable knowledge while preserving benign capabilities.

By Bocheng Ju, Jianhua Wang, Chengliang Liu, Xiaolin Chang

llms fine-tuning benchmarks safety

Related stories

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning