arXiv Machine Learning

Simple Meta-Learning for HPO and CASH Using Performance Representations

The paper proposes a new meta‑learning representation for hyperparameter optimization (HPO) and combined algorithm selection and hyperparameter optimization (CASH) that encodes each configuration by its historical performance across prior tasks, rather than by explicit hyperparameter values. This performance‑based representation can be used with standard surrogate models, such as linear Bayesian ridge regression, and is applied to a diversity‑based re‑ranking criterion (maximal marginal relevance) as a posterior‑free acquisition rule. Experiments on the TabArena benchmark show that this approach outperforms dedicated meta‑learning baselines, achieving comparable tuned performance in about half the number of trials, and remains competitive for single‑model HPO while scaling independently of search‑space dimensionality.

arXiv Machine Learning

Oct 15

Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH

The paper introduces LB-MCTS, a trajectory-structured optimization framework that combines Monte Carlo Tree Search, Bayesian Optimization, and Large Language Models to address the CASH problem in AutoML. It uses a shared tree state for algorithm selection, hyperparameter refinement, and synergy between BO and LLM proposals, with a reliability-aware policy that shifts from LLM-driven to BO-driven suggestions as the surrogate model improves. Experiments on 104 AMLB datasets show that LB-MCTS consistently outperforms existing BO-based, LLM-based, and hybrid baselines.

By Beicheng Xu, Weitong Qian, Lingching Tung, Yupeng Lu, Bin Cui

llms

arXiv AI

Oct 31

Rethinking Bayesian Optimization for Co-Optimizing LLM Training Configurations

The paper introduces JoBS, a Bayesian optimization framework that co‑optimizes data and model configurations for large language model training without requiring full training runs at each iteration. JoBS first allocates part of the budget to train a scaling‑law‑inspired performance predictor using only a few training steps, then uses this predictor to guide subsequent BO iterations, thereby exploring many more configurations within the same budget. The authors analyze JoBS’s regret, derive optimal budget splits, and demonstrate that it outperforms both independent optimization and existing multi‑fidelity BO baselines across diverse downstream tasks.

By Zhiliang Chen, Alfred Wei Lun Leong, Shao Yong Ong, Apivich Hemachandra, Gregory Kang Ruey Lau, Chuan-Sheng Foo, Zhengyuan Liu, Nancy F. Chen, Bryan Kian Hsiang Low

llms fine-tuning

arXiv Machine Learning

Oct 7

Direct Regret Optimization in Bayesian Optimization

arXiv:2507.06529v2 Announce Type: replace Abstract: Bayesian optimization (BO) is a powerful paradigm for optimizing expensive black-box functions. Traditional BO methods typically rely on separate h...

By Fengxue Zhang, Yuxin Chen

llms benchmarks

arXiv Machine Learning

Sep 11

Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates

arXiv:2604. 13130v2 Announce Type: replace Abstract: We study learning to learn through the lens of hyperparameter tuning.

By Saumya Goyal, Rohith Rongali, Ritabrata Ray, Barnab\'as P\'oczos

arXiv AI

Oct 21

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

arXiv:2610.27895v1 Announce Type: cross Abstract: Direct preference optimisation (DPO) is widely used for language model post-training, but its performance depends strongly on the choice of its hyper...

By Hannan Mahadik, David Salinas, Jacek Golebiowski, Aaron Klein

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

arXiv:2510. 06048v4 Announce Type: replace Abstract: Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks.

By Jie Hao, Rui Yu, Wei Zhang, Huixia Wang, Jie Xu, Mingrui Liu

llms benchmarks

arXiv AI

Oct 27

Automatic Configuration of LLM Post-Training Pipelines

arXiv:2603.18773v2 Announce Type: replace-cross Abstract: LLM post-training pipelines that combine supervised fine-tuning and reinforcement learning are difficult to configure under realistic compute...

By Channe Chwa, Xinle Wu, Yao Lu

llms reinforcement-learning fine-tuning

arXiv AI

Jun 9

Scaling Decision-Focused Learning to Large Problems with Lagrangian Decomposition

arXiv:2606. 08797v1 Announce Type: cross Abstract: Decision-focused learning has shown great promise for addressing predict-then-optimize problems, particularly in the presence of under-specified models.

By St\'ephane Eilles-Chan Way, Hugo Percot, Quentin Cappart, Tias Guns, Louis-Martin Rousseau

benchmarks

arXiv Machine Learning

Jun 30

The Voronoi Bottleneck: Capacity-Aware Dense Retrieval for Product Search

arXiv:2606. 28359v1 Announce Type: cross Abstract: Dense embedding retrieval compresses all relevance information into a single inner product, imposing a fundamental geometric limit -- the Voronoi Bottleneck -- on the number of query-document relevance patterns expressible at fixed embedding dimension (d).

By Charith Chandra Sai Balne, Rithwik Maramraju, Siddharth Pratap Singh, Rohit Upadhyay, Aditya Singh, Chittaranjan Tripathy, Yogananda Domlur Seetharama

rag

arXiv Machine Learning

Aug 18

Decision-Driven Regularization: A Blended Model for Learning and Optimization

arXiv:2608. 15124v1 Announce Type: new Abstract: In contextual optimization, the decision-maker seeks optimal decisions to minimize a cost function, that varies based on observed features.

By Gar Goei Loke, Qinshen Tang, Yangge Xiao, Xun Zhang

benchmarks

arXiv AI

Sep 25

DeGRe: Dense-supervised Generative Reranking for Recommendation

DeGRe is a dense‑supervised generative reranking framework designed to improve multi‑stage recommender systems by addressing label bias and credit assignment issues. It uses an offline Lookahead Evaluator with beam search to generate dense supervision signals, which are distilled into a lightweight Online Generator that can perform efficient greedy decoding at inference time. Experiments show that DeGRe outperforms baselines on public benchmarks and industrial datasets, and it has been successfully deployed on Taobao Flash Shopping to enhance online recommendations.

By Chaotian Song, Jingyao Zhang, Chenghao Chen, Zisen Sang, Dehai Zhao, Guodong Cao, Boxi Wu, Deng Cai, Jia Jia

rag benchmarks safety

arXiv AI

Jul 14

Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

arXiv:2607. 09739v1 Announce Type: new Abstract: We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite.

By Jihan Yao, Gantavya Bhatt, Arnav Das, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhardwaj, Chang Su, Jialei Wang, Yikai Zhu, Sugam Devare, Damon Mosk-Aoyama, Zhen Dong, Venkat Krishna Srinivasan, Yineng Zhang, Oleksii Kuchaiev, Jiantao Jiao, Banghua Zhu, Jeff Bilmes

llms rag benchmarks

Related stories