arXiv Machine Learning June 16, 2026 By Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia, Leonidas Guibas, Luiz Velho, Tiago Novello

Functional Gradient Descent with Adaptive Representations

Read the original on arXiv Machine Learning →

arXiv:2606. 16926v1 Announce Type: cross Abstract: Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

computer-vision

arXiv Machine Learning

Aug 4

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.

By John Chiang

arXiv Machine Learning

Aug 13

A New First-Order Meta-Learning Algorithm with Convergence Guarantees

arXiv:2409. 03682v2 Announce Type: replace Abstract: Learning new tasks by leveraging prior experience is a fundamental trait of intelligent systems.

By El Mahdi Chayti, Martin Jaggi

llms efficiency safety

arXiv Machine Learning

Jul 30

Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction

arXiv:2607. 26562v1 Announce Type: cross Abstract: We study optimization under performative prediction, where deploying a model affects the future data distribution.

By Hiroki Hamaguchi, Yuya Hikima, Hiroshi Sawada, Akiko Takeda

arXiv Machine Learning

Aug 11

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.

By Binchuan Qi

safety

arXiv Machine Learning

Aug 26

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...

By Tim Tsz-Kit Lau, Han Liu, Mladen Kolar

arXiv Machine Learning

1d ago

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.

By Cristian McGee, El Houcine Bergou, Aritra Dutta

llms fine-tuning

One email a morning, machine-written

Related stories

Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients

A New First-Order Meta-Learning Algorithm with Convergence Guarantees

Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning