arXiv Machine Learning November 2, 2026 By Reza Shirkavand, Peiran Yu, Qi He, Heng Huang

Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training

Read the original on arXiv Machine Learning →

Bilevel-ZOFO is a bilevel optimization framework that combines fast, low‑memory first‑order (FO) parameter‑efficient fine‑tuning (PEFT) in the inner loop with stable, memory‑efficient zeroth‑order (ZO) updates of the full model in the outer loop. This design reduces the variance of ZO gradient estimates, stabilizes training, and mitigates prompt sensitivity while preserving the full model’s capacity. Empirical results show Bilevel-ZOFO achieves 2–4× faster training than existing ZO and FO‑PEFT methods while maintaining similar memory efficiency, and it functions as an efficient meta‑learning algorithm for rapid few‑shot adaptation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 11

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.

By Feihu Jin, Shipeng Cen, Ying Tan

llms fine-tuning benchmarks

arXiv AI

Oct 15

StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

arXiv:2610.16451v1 Announce Type: cross Abstract: Fine-tuning large language models at scale is increasingly limited by the memory cost of backpropagation. Zeroth-Order (ZO) optimizers reduce this bu...

By Xuanlin Yang, Jia Li, Zhouchen Lin

llms fine-tuning

arXiv Machine Learning

Jun 16

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.

By Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin, Aleksandr Beznosikov

llms fine-tuning

arXiv Machine Learning

Jun 5

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.

By Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin, Deliang Fan, Li Yang

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 9

Efficient Long-Horizon Learning for Learned Optimization

arXiv:2607. 06772v1 Announce Type: new Abstract: Learned optimization aims to improve upon hand-designed optimizers (e.

By Xiaolong Huang, Benjamin Th\'erien, James Harrison, Eugene Belilovsky

llms computer-vision

Hugging Face Trending Papers

Jul 7

Efficient Long-Horizon Learning for Learned Optimization

Learned optimization aims to improve upon hand-designed optimizers (e. g.

llms computer-vision

One email a morning, machine-written

Related stories

Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning

StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

Efficient Long-Horizon Learning for Learned Optimization

Efficient Long-Horizon Learning for Learned Optimization