arXiv AI
Always-On Experimentation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
5d ago
Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments
The paper explores how data from fixed A/B tests can guide the deployment of adaptive experiments using contextual bandits. By combining off‑policy evaluation with a controlled warm‑start simulation, the authors rank pre‑specified adaptive and non‑adaptive policies using doubly robust estimators. Experiments on synthetic trials and real benchmarks show that adaptive, context‑aware policies outperform fixed allocations when heterogeneity exists, but offer little advantage otherwise.
arXiv Machine Learning
Jul 17
Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap
arXiv:2607. 14604v1 Announce Type: new Abstract: Online controlled experiments are the gold standard for hypothesis testing in online platforms.
arXiv AI
Jun 15
CARE: Controlling LLM-Generated Policies through Auditable Review of Evidence in Scientific Experimentation
arXiv:2606. 14581v1 Announce Type: cross Abstract: Granting LLMs direct control over costly, irreversible scientific experiments leads to unsafe exploration and unstable performance, but discarding LLM creativity entirely sacrifices significant optimization potential.
arXiv Machine Learning
Jul 13
Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation
arXiv:2506. 03062v2 Announce Type: replace Abstract: A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.
arXiv Machine Learning
Jul 13
Global Sequential Testing for Multi-Stream Auditing
arXiv:2602. 21479v3 Announce Type: replace-cross Abstract: Across many risk-sensitive areas, it is critical to continuously audit machine learning systems as we receive more data to quickly determine if they are performing as designed.
arXiv AI
Aug 11
Agentic Auto-Research is Fuzz Testing
arXiv:2608. 09855v1 Announce Type: new Abstract: Autonomous research agents can generate experiments faster than researchers can validate them.