The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.
By Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate
The paper introduces the concept of Rashomon Representation, describing how different foundation models can encode the same input in substantially different ways, leading to representational disagreement. It proposes a lightweight predictor that, using a single model’s representation, estimates whether an input will align with or diverge from other models’ representations, based on consistent input-dependent patterns. Experiments across various models and datasets demonstrate that this disagreement is predictable and can be used for efficient reliability estimation of foundation models.
By Mingyue Ma, Zongbo Han, Changqing Zhang, Guangyu Wang
Foundation models are increasingly adopted across a wide range of applications, often serving as core blocks within AI systems. Yet different foundation models may encode the same input from multiple...
arXiv:2605. 28021v2 Announce Type: replace Abstract: Out-of-distribution (OOD) detection is essential for deploying machine learning models in open-world and safety-critical scenarios, where test inputs may deviate from the training distribution and overconfident predictions on unknown samples can lead to unreliable decisions.
By Fengqiang Wan, Qing-Yuan Jiang, Fu Shen, Yang Yang
arXiv:2605.24818v3 Announce Type: replace-cross
Abstract: The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our c...
By Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole, Robin Jia
arXiv:2605. 24818v2 Announce Type: replace-cross Abstract: The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored.
By Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole, Robin Jia
The paper audits the confidence outputs of seven feed‑forward 3D reconstruction backbones across 13 datasets, evaluating four properties: error ranking, average error‑to‑uncertainty ratio, slope of this ratio, and coverage of the implied error distribution. While confidence ranks errors well, the decoded uncertainty is consistently too small—off by at least 2.4× on median cases—and worsens with higher confidence. A post‑hoc power‑law fit per backbone‑dataset pair improves all four metrics at the dataset level, reducing the median error by 1.35×, but fails to correct coverage for many held‑out scenes, indicating the models lack the correct error scale and distribution shape.
By Nanxing Nick Deng, Qing Cheng, Niclas Zeller, Daniel Cremers
arXiv:2610.16152v1 Announce Type: cross
Abstract: Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination an...
By Hengxiang Zhang, Junxian Liu, Hongxin Wei
The paper introduces Calibration-Aware Uncertainty Cascades (CAUC), a post‑hoc framework that calibrates each model’s confidence independently and uses these calibrated scores to decide when to accept an early prediction, invoke a stronger model, or combine outputs. CAUC establishes a common reliability scale across heterogeneous models, decoupling deployment policies from specific model pools or budgets. Experiments on six language benchmarks show a 1.9% relative accuracy gain over strong‑model‑only inference while cutting strong‑model calls by about 47%, and on image classification it maintains or improves performance while reducing GFLOPs by up to 57%.
By Yilin Zhang, Han Jiang, Cai Xu, Ying Liu, Wei Zhao
The paper introduces Conditional Accuracy Profiling (CAP), a diagnostic framework that breaks down LLM‑judge accuracy into eight deployment conditions covering content sensitivity, robustness, and rationale quality. CAP is benchmark‑agnostic and can be applied directly when annotations are available, via task‑subset proxies, or through controlled augmentation. Applied to seven judges across six benchmarks, CAP reveals that aggregate accuracy masks important differences—for example, judges highly sensitive to omitted qualifications rank poorly on a Hard‑Constructed subset, and Position Robustness is generally stable but can degrade sharply under adversarial stress.
By Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li
arXiv:2608. 05670v1 Announce Type: new Abstract: A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness.
By Rasul Khanbayov, Hasan Kurban
arXiv:2601. 07965v2 Announce Type: replace Abstract: When a model knows when it does not know, many possibilities emerge.
By Chenjie Hao, Weyl Lu, Yuko Ishiwaka, Zengyi Li, Weier Wan, Yubei Chen