The paper introduces Bayesian Optimization (BO) techniques that incorporate rich auxiliary information—such as training curves, expert notes, images, and prior knowledge—using large language models (LLMs). Three new methods are proposed to integrate this auxiliary data into BO, and they are evaluated on hyperparameter optimization benchmarks and a real-world nuclear fusion task. The results show that these LLM-enhanced BO methods consistently outperform standard BO and existing LLM-based optimization approaches.
arXiv:2608. 10867v1 Announce Type: new Abstract: Gradient-free post-training has emerged as a compelling alternative to gradient-based optimization for large language models (LLMs), but existing approaches remain costly.
arXiv:2608. 04113v1 Announce Type: cross Abstract: Black-box optimization is a ubiquitous problem in science and engineering, often dealing with expensive objective functions with cheaper lower-fidelity proxies available.
arXiv:2211. 14411v5 Announce Type: replace-cross Abstract: Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as on memory usage or latency, on top of the performance requirement.
GRAPE is a two‑stage Bayesian optimization framework that first refines the local gradient posterior using a closed‑form acquisition function and then selects update directions by maximizing expected decrease conditioned on descent. The authors prove that the refinement stage monotonically reduces local uncertainty and that the progress‑aware direction converges to true steepest descent as the posterior sharpens. Empirical results show GRAPE achieves a 5.4× speedup on black‑box adversarial attacks and reduces final average regret by 3.8 log‑units on large language model prompt‑optimization tasks.
arXiv:2511. 02570v3 Announce Type: replace Abstract: Bayesian optimization (BO) is a widely used approach to hyperparameter optimization (HPO).
arXiv:2606. 07841v1 Announce Type: cross Abstract: Black-box variational inference (BBVI) is a methodology for posterior approximation that relies on stochastic optimization.
arXiv:2607. 00865v1 Announce Type: new Abstract: Bayesian Optimisation (BO) under unknown constraints is particularly challenging when feasible regions are small.
The paper presents a method that uses Constrained Bayesian Optimization (CBO) to minimize the energy consumption of machine learning models while ensuring their generalization performance stays above a specified threshold. By treating energy usage as the primary objective and performance as a constraint, the authors demonstrate that CBO can reduce training energy costs on both regression and classification tasks without sacrificing predictive accuracy.
The paper introduces Power‑Law Entropy Search (PLES), a computational‑cost‑aware acquisition function that uses multi‑fidelity Bayesian optimization to efficiently estimate optimal hyperparameter scaling laws for large language model training. PLES focuses on reducing the overall uncertainty of scaling law estimates rather than optimizing a single objective, selecting configurations that maximize uncertainty reduction per unit computational cost. Experiments on synthetic benchmarks, surrogate models, and real LLM pre‑training runs show that PLES converges to accurate scaling laws using less than one‑tenth of the computational budget required by conventional grid search and other baselines.
arXiv:2603. 29730v2 Announce Type: replace-cross Abstract: We present mlr3mbo, a modular toolbox for Bayesian optimization in R.
arXiv:2605.00068v2 Announce Type: replace-cross Abstract: Inertial Confinement Fusion (ICF) holds transformative promise for sustainable, near-limitless clean energy, yet remains constrained by prohi...
arXiv:2609.36788v1 Announce Type: new Abstract: Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian...