The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.
By Cristian McGee, El Houcine Bergou, Aritra Dutta
arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.
By Arzu Ahmadova, Ismail Huseynov
arXiv:2604. 14669v2 Announce Type: replace Abstract: Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored.
By Minhak Song, Liang Zhang, Bingcong Li, Niao He, Michael Muehlebach, Sewoong Oh
arXiv:2610.07290v1 Announce Type: cross
Abstract: Recent advances in penalty-based methods for stochastic bilevel optimization (SBO) have eliminated the need for second-order derivative oracles. Howe...
By Xingyu Chen, Ming Yang, Quanqi Hu, Tianbao Yang
arXiv:2504.09409v3 Announce Type: replace-cross
Abstract: In this paper, we study nonconvex constrained stochastic zeroth-order optimization problems with exact constraints and stochastic objective e...
By Qiankun Shi, Han Yuan, Xiao Wang, Hao Wang
arXiv:2610. 03222v1 Announce Type: cross Abstract: This paper studies the complexity of convex optimization using lazy second-order oracles (Doikov, Chayti, and Jaggi, ICML 2023), where an algorithm queries gradients every iteration and Hessians once per $m$ iterations.
By Xinliang Zhang, Lesi Chen, Chengchang Liu, Jingzhao Zhang
arXiv:2509. 14969v2 Announce Type: replace Abstract: We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning.
By Jean-Fran\c{c}ois Aujol, J\'er\'emie Bigot, Camille Castera
arXiv:2608. 10418v1 Announce Type: cross Abstract: Recent work has shown that, for smooth convex optimization, plain gradient descent can be accelerated from its textbook convergence rate of $O(T^{-1})$ (where $T$ denotes the number of iterations) to $O\big(T^{-\log_2(1+\sqrt{2})}\big)$ using carefully designed stepsize schedules alone, without resorting to momentum or other algorithmic modifications.
By Jianhao Ma, Yuxin Chen
arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.
By John Chiang
arXiv:2608. 12009v1 Announce Type: cross Abstract: Bregman proximal stochastic gradient (BPSG) methods bring variance-reduced composite optimization to objectives whose geometry is poorly captured by Euclidean smoothness.
By Chenhan Jin, Shengze Xu, Binghui Xie, Kaiwen Zhou, Fan Jia, James Cheng, Tieyong Zeng
arXiv:2606. 01764v1 Announce Type: cross Abstract: We revisit the convergence guarantees of the Extragradient (EG) method for unconstrained biaffine min-max optimization.
By Yue Wu, Weiqiang Zheng, Yang Cai, Haipeng Luo
arXiv:2608. 12665v1 Announce Type: cross Abstract: For solving nonconvex equality-constrained optimization problems, a recent Gradient-Eigenstep Algorithm by Goyens et al.
By Frank E. Curtis, Lingjun Guo, Daniel P. Robinson