← Back to all news
arXiv Machine Learning By Emiel Witting, Cheng Yan, Tom Julian Viering

Simple Meta-Learning for HPO and CASH Using Performance Representations

Read the original on arXiv Machine Learning →

The paper proposes a new meta‑learning representation for hyperparameter optimization (HPO) and combined algorithm selection and hyperparameter optimization (CASH) that encodes each configuration by its historical performance across prior tasks, rather than by explicit hyperparameter values. This performance‑based representation can be used with standard surrogate models, such as linear Bayesian ridge regression, and is applied to a diversity‑based re‑ranking criterion (maximal marginal relevance) as a posterior‑free acquisition rule. Experiments on the TabArena benchmark show that this approach outperforms dedicated meta‑learning baselines, achieving comparable tuned performance in about half the number of trials, and remains competitive for single‑model HPO while scaling independently of search‑space dimensionality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 15

Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH

The paper introduces LB-MCTS, a trajectory-structured optimization framework that combines Monte Carlo Tree Search, Bayesian Optimization, and Large Language Models to address the CASH problem in AutoML. It uses a shared tree state for algorithm selection, hyperparameter refinement, and synergy between BO and LLM proposals, with a reliability-aware policy that shifts from LLM-driven to BO-driven suggestions as the surrogate model improves. Experiments on 104 AMLB datasets show that LB-MCTS consistently outperforms existing BO-based, LLM-based, and hybrid baselines.

By Beicheng Xu, Weitong Qian, Lingching Tung, Yupeng Lu, Bin Cui
llms
More like this →
arXiv AI
Oct 31

Rethinking Bayesian Optimization for Co-Optimizing LLM Training Configurations

The paper introduces JoBS, a Bayesian optimization framework that co‑optimizes data and model configurations for large language model training without requiring full training runs at each iteration. JoBS first allocates part of the budget to train a scaling‑law‑inspired performance predictor using only a few training steps, then uses this predictor to guide subsequent BO iterations, thereby exploring many more configurations within the same budget. The authors analyze JoBS’s regret, derive optimal budget splits, and demonstrate that it outperforms both independent optimization and existing multi‑fidelity BO baselines across diverse downstream tasks.

By Zhiliang Chen, Alfred Wei Lun Leong, Shao Yong Ong, Apivich Hemachandra, Gregory Kang Ruey Lau, Chuan-Sheng Foo, Zhengyuan Liu, Nancy F. Chen, Bryan Kian Hsiang Low
llmsfine-tuning
More like this →