arXiv Statistics ML

What Fixed-Rollout pass@k Evaluations Can Identify

arXiv Computer Vision
Oct 28

How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop

The paper introduces a denoised counterfactual protocol to accurately measure the selectable headroom of world action models (WAMs) and uses these measurements to build a fail‑gated selector. Across four WAMs and two benchmarks, the study shows that the standard estimate overstates selectable headroom, that structured action perturbations can add up to 16 points of headroom, and that imagined futures retain candidate‑outcome information without loss. The selector improves closed‑loop task success from 76.9 % to 81.6 % on held‑out training episodes and from 62.6 % to 69.8 % on unseen tasks.

By Haojia Gao, Yue Cao, Yumin Liu, Haohua Que, Mingkai Liu, Guanqiu Guo, Zhuo Zou, Bokui Chen, Xueqian Wang
arXiv AI
Sep 17

Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents

The paper introduces XConf, an experiential confidence estimator that augments a language model’s current inference with a record of its past graded episodes. By recalling similar past tasks and reflecting on past outcomes, XConf generates confidence scores without accessing logits or updating weights, achieving superior discrimination and calibration across diverse benchmarks. The method demonstrates significant gains in selective prediction, improving success rates on agent tasks by up to 8.7 points.

By Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Yulong Chen, Dharshan Kumaran, Nigel Collier
arXiv AI
1d ago

Model-Free Assessment of Simulator Fidelity via Quantile Curves

The paper introduces a model‑free method for assessing how well simulators match real‑world systems by constructing confidence sets for latent population parameters across different scenarios. It then uses these sets to estimate a quantile function that provides a distribution‑level risk profile of the simulator, enabling statistical inference, risk measures such as CVaR, and comparisons between simulators. The approach is agnostic to model type and applicable to diverse output spaces, and is demonstrated on four large language models using the WorldValueBench dataset.

By Garud Iyengar, Yu-Shiou Willy Lin, Kaizheng Wang
arXiv AI
6d ago

Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text

The paper investigates AI‑assisted measurement for administrative records where a single text label applies to many records and the target is a monetary total. Using 121,070 grant‑purpose texts from US tax filings, the authors compare rule‑based and language‑model classifiers, evaluate fixed‑predictor and cross‑fitted estimators under Poisson sampling, and test dollar‑weighted error rules and a two‑phase review policy. Results show that a calibrated two‑phase policy can reduce RMSE by 28% with a limited budget, while independent human readers shift totals by billions and institutional intervals under‑cover.

By George Koomullil (Ascendr, PBC)
arXiv AI
Aug 28

Invocation-Level Reliability of Tool-Using Agents

The paper investigates the reliability of tool‑using agents, focusing on two failure modes: selecting the wrong tool and constructing incorrect arguments. It introduces a correct‑invocation rate metric to distinguish these errors and evaluates five open‑weight models on multi‑step tasks up to depth 8, finding that by depth 6 about 70% of a model’s clean‑context capability is lost due to earlier mistakes. The study reveals that exact‑match scoring against a fixed gold trajectory forces severity and recovery parameters to extreme values, and proposes a conditional‑on‑state scoring remedy that yields more realistic severity estimates.

By Afiya Noorain, Subhranshu Mohanty, Amritesh Banerjee, Abhijit Dasgupta