arXiv:2609. 18622v1 Announce Type: new Abstract: Classifiers can make identical predictions yet require labels to compare their selective performance: confidence ranks weight the same errors differently.
arXiv:2412. 12807v4 Announce Type: replace-cross Abstract: Selective classification is a powerful tool for automated decision-making in high-risk scenarios, allowing classifiers to act only when confident and abstain when uncertainty is high.
Classifiers can make identical predictions yet require labels to compare their selective performance: confidence ranks weight the same errors differently. We quantify this requirement for the area und...
arXiv:2607. 03999v1 Announce Type: cross Abstract: Estimating heterogeneous treatment effects (CATE) requires simultaneously detecting effect modification and quantifying estimation uncertainty.
arXiv:2607. 18088v1 Announce Type: new Abstract: Standard evaluation of many recognition systems contains distribution shift by construction, since benchmarks place disjoint conditions in the training and test splits.
arXiv:2607. 24562v1 Announce Type: new Abstract: Large language models serve heterogeneous populations structured by domain, topic difficulty, and linguistic style.
arXiv:2404. 02141v5 Announce Type: replace-cross Abstract: In both observational data and randomized control trials, researchers select statistical models to articulate how the outcome of interest varies with combinations of observable covariates.
The paper introduces a method to certify selective prediction in machine learning systems by computing the availability of safety gates through exact-binomial inversion and dynamic programming. It demonstrates that a truth-informed planner can significantly improve mean coverage over naive approaches, and that reallocating error budgets further enhances coverage across diverse applications such as LLM tool‑calling, content moderation, lesion classification, and recommendation. The study highlights the importance of planning and finite‑sample estimation in ensuring reliable, granular deployment of selective predictors.
The paper derives the exact finite‑population variance of a weighted risk estimator for rare‑event forecasting in dependent sequences and solves for the optimal stratified allocation of a small subsample. It shows that the optimal allocation is equal across strata, independent of the imbalance ratio, and provides a parameter‑free efficiency prediction A(π,f). The authors validate these theoretical predictions on a real‑world dataset of U.S. equities, demonstrating that the predicted ordering of sampling designs matches empirical results.
The paper introduces a new estimand for conditional distributional treatment effects that captures how treatments influence the entire outcome distribution, including variance and tail risks, in a covariate-dependent manner. It presents a doubly robust estimator that is minimax optimal locally and uses it to construct a test for global homogeneity of conditional potential outcome distributions. The test accommodates discrepancies beyond the maximum mean discrepancy, guarantees valid type‑1 error, is consistent against fixed alternatives, and includes a computationally efficient, permutation‑free algorithm with exact closed‑form expressions for two natural discrepancies.
arXiv:2609.09245v1 Announce Type: new Abstract: Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/ran...
arXiv:2607. 08347v1 Announce Type: cross Abstract: Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled.