The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.
arXiv:2606. 02198v1 Announce Type: new Abstract: Prediction tasks over individual futures, which are inherently noisy, often admit multiple similarly accurate models.
The paper introduces the concept of Rashomon Representation, describing how different foundation models can encode the same input in substantially different ways, leading to representational disagreement. It proposes a lightweight predictor that, using a single model’s representation, estimates whether an input will align with or diverge from other models’ representations, based on consistent input-dependent patterns. Experiments across various models and datasets demonstrate that this disagreement is predictable and can be used for efficient reliability estimation of foundation models.
AdaFuse is an adaptive ensemble decoding framework for large language models that dynamically selects fusion units during generation. It uses an uncertainty-based criterion to decide when to ensemble, applying a diversity-aware scaling strategy in uncertain states while continuing direct generation when confident. Experiments on question answering, arithmetic reasoning, and machine translation show AdaFuse outperforms strong baselines with an average relative improvement of 6.88%.
arXiv:2607. 26368v1 Announce Type: cross Abstract: Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in qualitatively different ways.
The paper introduces Alignment Forecasting, a method for predicting whether fine‑tuning a language model on a given dataset will increase specific alignment failures such as deception or sycophancy. It presents ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions across many models, datasets, and failure modes, and shows that a simple forecasting scaffold using an LLM’s assessment of dataset bias can outperform baseline forecasters. The authors demonstrate that filtering out high‑risk training examples identified by the forecaster can improve alignment in multiple‑choice evaluations, though benefits in open‑ended conversations remain uncertain.
Foundation models are increasingly adopted across a wide range of applications, often serving as core blocks within AI systems. Yet different foundation models may encode the same input from multiple...
arXiv:2610.01428v1 Announce Type: cross Abstract: Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed...
The paper investigates how large language models (LLMs) used as judges in absolute scoring tasks exhibit systematic biases that compromise reliability. It shows that a judge’s task accuracy strongly predicts both its judging accuracy and its directional bias, yet more capable examinee models consistently receive more lenient judgments. To mitigate these biases, the authors propose a calibrated weighted majority voting (WMV) ensemble that estimates judges’ error rates from inter-judge agreement patterns, achieving near-oracle performance without labeled data and improving both accuracy and fairness.
arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.
arXiv:2512. 22240v5 Announce Type: replace-cross Abstract: Machine learning models are primarily judged by predictive performance, especially in applied genomics, where explanations are read as biological findings.
arXiv:2601. 09071v2 Announce Type: replace Abstract: The existence of multiple, equally accurate models for a given predictive task leads to predictive multiplicity, where a Rashomon set of models achieve similar accuracy but diverge in their individual predictions.
arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.