arXiv:2608. 07914v1 Announce Type: new Abstract: Behavioral contamination detectors can return "no evidence" either because a benchmark is clean or because the audit has little power.
By Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma
The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.
By Yigit Utku Bulut
The paper introduces a hypothesis testing framework for adaptive auditing of AI systems, addressing the challenge of limited, opportunistically collected data. It defines two competing null hypotheses—one from the model’s perspective and one from the auditor’s—and uses Safe Anytime-Valid Inference (SAVI) to perform betting-based tests that yield simultaneous e-processes. The authors prove that, with a powerful auditor, the two hypotheses become asymptotically inverse, allowing a global robustness certification, and demonstrate empirically that their methods control Type‑I error and can reach conclusions with as few as 20 observations.
By Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman, Yifan Mai, Jean Feng
The paper introduces two scaling laws that predict the performance of safety probes—tools used to monitor language models for harmful behavior—based on model size and the number of labeled examples. By creating six large safety datasets and training probes across multiple model families, the authors show that probe error decreases as a power law with model size and further improves with more examples, with the improvement rate increasing for larger models. These laws enable researchers to forecast probe accuracy and decide whether to invest in more data or a larger model to achieve a desired error threshold.
By Yuchen Wu (University of Washington), Jerry Wei (Anthropic)
The paper examines how to quantify the time AI tools save, contrasting a simple stopwatch approach with a more nuanced ledger view that accounts for errors introduced. It introduces model‑free metrics—time saved, excess escaped defects, and a break‑even defect cost—and applies them to a baseline checking model, randomized studies, and a replication of METR’s developer data. The author also reports on a small experiment with AI reviewers, showing that reduced effort can lead to faster but more error‑prone outcomes.
By Solomon Shalom Lijo
arXiv:2610.41970v1 Announce Type: new
Abstract: World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improv...
By Jun Liu, Dajie Sun, Zhenglun Kong, Zhenjiang Fan, Peiyan Dong, Fan Yang, Wenbin Zhangg, Geng Yuan, Xue Lin, Yanzhi Wang, Dong Huang