arXiv Machine Learning

Bridging Spherical Black-Box Optimizers

arXiv:2606. 25761v1 Announce Type: new Abstract: When gradient information is unavailable, black-box optimization (BBO) methods provide a practical alternative.

arXiv Machine Learning
1d ago

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.

By Cristian McGee, El Houcine Bergou, Aritra Dutta
arXiv Machine Learning
Aug 5

Exploiting Separability in Multi-Scale Grey-Box Bayesian Optimization

arXiv:2608. 03045v1 Announce Type: new Abstract: We consider grey-box optimization problems where the decision variables naturally partition into black-box variables (as arguments to an expensive black-box function) and white-box variables, governed by a set of explicit, closed-form equations that also depend on the output of the black-box function.

By Joshua E. Hammond, Tyler A. Soderstrom, Brian A. Korgel, Michael Baldea
arXiv AI
Jul 7

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

arXiv:2607. 04033v1 Announce Type: cross Abstract: Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented.

By Siyuan Li, Jiabao Pan, Yumou Liu, Zhuoli Ouyang, Xin Jin, Xinglong Xu, Jingxuan Wei, Shengye Pang, Jintao Che, Xuanhe Zhou, Conghui He, Cheng Tan