arXiv Machine Learning
The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Statistics ML
Oct 7
When does conformal calibration need censoring weights? Cause-of-failure prediction sets under competing risks
The paper investigates when conformal calibration for cause‑of‑failure prediction sets under competing risks requires censoring weights. It shows that using only complete‑case calibration can lead to coverage that is either above or below the nominal level, especially when a substantial proportion of subjects are event‑free at the prediction horizon. The authors demonstrate through simulations that incorporating weights from a correctly specified censoring model restores nominal coverage, and they provide a finite‑sample lower bound that penalises censoring‑model error, highlighting the risk of under‑coverage when the model is misspecified.
arXiv Machine Learning
Oct 8
Certified by Abstention: Distribution-Free Guarantees for Chain-of-Thought Verifiers at Small Calibration Budgets
arXiv:2610.09541v1 Announce Type: cross Abstract: Signals that predict whether a chain-of-thought (CoT) trace is correct are compared by AUC, but deploying one requires a threshold with a guarantee....
arXiv AI
Oct 1
Hard-Gate Candidacy in a Deployed Validator Suite
The paper evaluates hard‑gate candidacy for validators in a deployed generative‑agent system by measuring how well each validator’s firing separates successful from failed builds. Across 13 validators and thousands of builds, only a few checks show statistically significant separation, while many fail to distinguish or never fire. The study highlights that skipped checks are recorded as passes, limiting detectable failure rates and underscoring the need for clearer evaluation records.
arXiv AI
Oct 1
How Much Can Reliability Drift Under a Fixed Confidence Distribution?
arXiv:2609.38917v1 Announce Type: new Abstract: A classifier's conditional accuracy can change while its confidence distribution stays exactly the same. We study the worst-case movement of the reliab...
arXiv Machine Learning
Aug 26
Replicable Conformal Prediction
arXiv:2608.23638v1 Announce Type: cross Abstract: Two analysts who calibrate the same predictive model on independent samples will deploy different prediction sets every time, because the calibration...
arXiv Machine Learning
Aug 14
When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide
arXiv:2608. 12489v1 Announce Type: new Abstract: Organizations decide whom to treat under a budget and want to know what a targeting rule would have earned before deploying it.