arXiv Machine Learning July 13, 2026 By Qining Zhang, Tanner Fiez, Yi Liu, Wenyang Liu

Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation

Read the original on arXiv Machine Learning →

arXiv:2506. 03062v2 Announce Type: replace Abstract: A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

safety

arXiv Machine Learning

Jul 17

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

arXiv:2607. 14604v1 Announce Type: new Abstract: Online controlled experiments are the gold standard for hypothesis testing in online platforms.

By Olivier Jeunen

llms

arXiv AI

2d ago

Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

arXiv:2608. 14425v1 Announce Type: new Abstract: LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise.

By Toby D. Pilditch

llms benchmarks

arXiv Machine Learning

Jun 18

Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED

arXiv:2606. 18750v1 Announce Type: cross Abstract: A/B testing has become the gold standard for data-driven decision-making in large-scale online experimentation, providing critical guidance for feature launch, pricing optimization, and user experience enhancement.

By Yu Zhang, Bokui Wan, Yongli Qin, Jinyong Ma, Yifan Guo

arXiv Machine Learning

1d ago

CFR without Unbiasedness: Deterministic Guarantees for Persistent Public-Chance Schedules

arXiv:2608. 14761v1 Announce Type: cross Abstract: At a finite public-chance cut, counterfactual regret minimization (CFR) must choose how many outcomes to evaluate before each regret update.

By Jiaxing Guo, Lei Ye

arXiv Machine Learning

Jul 30

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

arXiv:2607. 26253v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal.

By Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes

reinforcement-learning

arXiv Machine Learning

Jul 10

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

arXiv:2607. 08522v1 Announce Type: new Abstract: The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation.

By Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, Leshem Choshen

multimodal benchmarks

Related stories

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED

CFR without Unbiasedness: Deterministic Guarantees for Persistent Public-Chance Schedules

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data