Sample-Adaptive Zeroth-Order Optimization (SAZO) is a memory‑efficient fine‑tuning method for large language models that learns sample importance online and allocates queries adaptively. It estimates importance using gradient‑norm information from directional differences and tracks evolving importance with per‑example exponential moving averages, requiring only O(N) additional scalar state. SAZO provably matches the leading convergence term of a minimum‑variance sampling oracle and achieves faster convergence than standard methods, with experiments showing up to 6× faster convergence to the same training loss.
arXiv:2606. 02857v1 Announce Type: cross Abstract: Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation.
arXiv:2601. 04710v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) achieves strong performance but is often limited by the memory overhead of backpropagation.
arXiv:2608. 15665v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates.
arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.
arXiv:2606. 14970v1 Announce Type: new Abstract: Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data.
arXiv:2607. 01125v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance.
arXiv:2512.00763v2 Announce Type: replace-cross Abstract: Adaptive and non-Euclidean optimizers often outperform Euclidean methods such as stochastic gradient descent (SGD) in language modeling by a...
arXiv:2604. 08564v2 Announce Type: replace-cross Abstract: Auto-regressive models (ARMs) have established a dominant paradigm in language modeling.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
arXiv:2601. 03043v4 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency.
The paper introduces U-LoRA, an adaptive version of Low‑Rank Adaptation that uses conditioned gating to generate token‑specific utilization coefficients for the low‑rank subspace. By coordinating these coefficients with sequence‑level context and applying a bias‑corrected EMA prior, U‑LoRA achieves more consistent adaptive patterns and reduces noise from batch fluctuations. Experiments on mathematical reasoning and natural language understanding tasks show that U‑LoRA matches or surpasses strong LoRA baselines while keeping the same parameter budget.