The paper explores how data from fixed A/B tests can guide the deployment of adaptive experiments using contextual bandits. By combining off‑policy evaluation with a controlled warm‑start simulation, the authors rank pre‑specified adaptive and non‑adaptive policies using doubly robust estimators. Experiments on synthetic trials and real benchmarks show that adaptive, context‑aware policies outperform fixed allocations when heterogeneity exists, but offer little advantage otherwise.
By Jo\~ao Victor Ferreira Alves, Eduardo Rocha Laurentino, Gustavo de Oliveira Kanno, Thiago Costa Rizuti da Rocha
The paper studies distributionally robust ranking and selection (DRR&S), where the goal is to identify the best alternative under input uncertainty by considering multiple plausible input distributions. It introduces the concept of sequential additivity, showing that efficient sampling should focus on a small, additive set of critical scenarios rather than a multiplicative number. The authors prove an algorithm‑independent lower bound on sampling, design an additive allocation (AA) procedure that meets this bound and achieves exponentially decreasing error probability, and extend the approach to a general additive allocation (GAA) framework that incorporates traditional R&S sampling rules.
By Zaile Li, Yuchen Wan, L. Jeff Hong
arXiv:2609.33180v2 Announce Type: replace-cross
Abstract: As recursive self-improvement (RSI) rapidly advances, reliable evaluation becomes critical for guiding adaptive search. RSI typically relies...
By Xiaojing Sun, Yuhan Zeng, Zihua She, Xiao Wang
Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty. We characterize that requirement on a fixed grid of $M$ tasks with $L$ binary paths per task under the hard budget $(M+t)K$, where each path costs at most $K$ responses or episodes.
arXiv:2607. 14604v1 Announce Type: new Abstract: Online controlled experiments are the gold standard for hypothesis testing in online platforms.
By Olivier Jeunen
arXiv:2609. 29140v1 Announce Type: new Abstract: Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty.
By Yezhou Cheng, Runjia Du, Zeming Liu, Qibai Chen, Hang Lyu, Yilan Wei, Yankai Zeng, Bojun Lin
The paper introduces Cost-Aware Sequential Hypothesis Testing (CASHT), where a decision-maker selects sensing actions with varying random costs to identify the true hypothesis under an average-error constraint while minimizing expected total cost. For fixed costs, the optimal expected total cost scales as Θ(log(1/δ)) and can be achieved by Multihypothesis Sequential Probability Ratio Test-based procedures. The authors extend the framework to random costs under ex-post and ex-ante revelation models, analyze when action cancellation reduces cost, and demonstrate through simulations that CA variants consistently lower total cost compared to classical methods.
By George Vershinin, Asaf Cohen, Omer Gurewitz
arXiv:2608. 14425v1 Announce Type: new Abstract: LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise.
By Toby D. Pilditch
arXiv:2609.38914v1 Announce Type: new
Abstract: Evaluating interactive agents is expensive. Agent behavior is stochastic, so reliability must be measured over repeated trials, but failures are rare a...
By Priyanath Maji, Spandan Ghose Chowdhury
arXiv:2608.16466v2 Announce Type: replace-cross
Abstract: Bayesian optimal experimental design (BOED) aims to collect informative data by optimizing an expected utility reflecting the goals of an exp...
By David Chen, Michael Evans, Xinwei Li, Prateek Bansal, David J. Nott
arXiv:2609.38695v1 Announce Type: cross
Abstract: Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conce...
By Ricardo J. Sandoval, David Arbour, Avi Feller, Michael I. Jordan
arXiv:2609.15309v1 Announce Type: new
Abstract: Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to s...
By Kaiyuan Liu, Qiuyang Mang, Bo Peng, Wenhao Chai, Hanchen Li, Shreyas Pimpalgaonkar, Luke Zettlemoyer, Alex Dimakis, Alvin Cheung