arXiv AI
StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 2
ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
arXiv:2607. 01125v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance.
arXiv Machine Learning
Jun 11
Steering the Noise: Turning Random Perturbations into Effective Descent for Memory-Efficient LLM Fine-Tuning
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
arXiv Machine Learning
Jun 16
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.
arXiv Machine Learning
4d ago
Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training
Bilevel-ZOFO is a bilevel optimization framework that combines fast, low‑memory first‑order (FO) parameter‑efficient fine‑tuning (PEFT) in the inner loop with stable, memory‑efficient zeroth‑order (ZO) updates of the full model in the outer loop. This design reduces the variance of ZO gradient estimates, stabilizes training, and mitigates prompt sensitivity while preserving the full model’s capacity. Empirical results show Bilevel-ZOFO achieves 2–4× faster training than existing ZO and FO‑PEFT methods while maintaining similar memory efficiency, and it functions as an efficient meta‑learning algorithm for rapid few‑shot adaptation.
arXiv Machine Learning
Jun 5
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.
arXiv AI
Jun 3
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
arXiv:2606. 02857v1 Announce Type: cross Abstract: Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation.