The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.
By Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate
The paper introduces the concept of Rashomon Representation, describing how different foundation models can encode the same input in substantially different ways, leading to representational disagreement. It proposes a lightweight predictor that, using a single model’s representation, estimates whether an input will align with or diverge from other models’ representations, based on consistent input-dependent patterns. Experiments across various models and datasets demonstrate that this disagreement is predictable and can be used for efficient reliability estimation of foundation models.
By Mingyue Ma, Zongbo Han, Changqing Zhang, Guangyu Wang
Foundation models are increasingly adopted across a wide range of applications, often serving as core blocks within AI systems. Yet different foundation models may encode the same input from multiple...
arXiv:2605. 28021v2 Announce Type: replace Abstract: Out-of-distribution (OOD) detection is essential for deploying machine learning models in open-world and safety-critical scenarios, where test inputs may deviate from the training distribution and overconfident predictions on unknown samples can lead to unreliable decisions.
By Fengqiang Wan, Qing-Yuan Jiang, Fu Shen, Yang Yang
arXiv:2605.24818v3 Announce Type: replace-cross
Abstract: The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our c...
By Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole, Robin Jia
arXiv:2605. 24818v2 Announce Type: replace-cross Abstract: The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored.
By Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole, Robin Jia