arXiv Computation and Language
Holdout Best-of-N: Unbiased Evaluation and Its Cost
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Jul 14
Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-$K$ Objective
arXiv:2607. 11146v1 Announce Type: new Abstract: We study the coupled objective J_K^WOR = E_{S ~ PL-WOR_K}[max_{i in S} R_i]: the expected maximum reward of a size-K Plackett-Luce draw without replacement, the law of Gumbel-Top-K / Stochastic Beam Search decoding.
arXiv Machine Learning
Oct 2
Regret Analysis of Retry-Based Bandits
arXiv:2605.20854v3 Announce Type: replace Abstract: We provide the first regret analysis of ReMax in stochastic multi-armed bandits. Originally introduced for reinforcement learning, ReMax is motivat...
arXiv Machine Learning
6d ago
Sharp Bounds on the Winner's Bias under Independent and Nonnegatively Correlated Noise
arXiv:2610.31948v1 Announce Type: new Abstract: Reporting the best of $M$ noisy, unbiased scores overstates the winner's value by up to $c\sqrt{V}$, where $V$ is the total noise variance and the shar...
arXiv Machine Learning
Jun 15
A Complexity Measure for Active Learning in Multi-group Mean Estimation
arXiv:2606. 14690v1 Announce Type: new Abstract: We study a \emph{max-risk} objective for active learning in a multi-group mean estimation $d$-armed bandits: a learner adaptively allocates a budget of $T$ samples across $d$ groups to minimize the worst-case uncertainty index $\max_{k\in[d]}\sigma_k^2/n_k$, where $\sigma_k$ is the standard deviation of the distribution of arm $d$, and $n_k$ is the number of times arm $d$ is sampled.
arXiv AI
Oct 1
Bandits with Multiple Optimal Arms: Minimax Regret and Non-Adaptivit
arXiv:2609. 38659v1 Announce Type: cross Abstract: We study multi-armed bandits (MAB) with multiple optimal arms, motivated by the fact that many practical decision making problems admit multiple correct answers.
arXiv AI
Jun 17
Optimism Stabilizes Thompson Sampling for Adaptive Inference
arXiv:2602. 06014v2 Announce Type: replace-cross Abstract: Thompson sampling (TS) is widely used for stochastic multi-armed bandits, yet its inferential properties under adaptive data collection are subtle.