← Back to all news
arXiv AI By Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

Read the original on arXiv AI →

The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 1

AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs

AdaFuse is an adaptive ensemble decoding framework for large language models that dynamically selects fusion units during generation. It uses an uncertainty-based criterion to decide when to ensemble, applying a diversity-aware scaling strategy in uncertain states while continuing direct generation when confident. Experiments on question answering, arithmetic reasoning, and machine translation show AdaFuse outperforms strong baselines with an average relative improvement of 6.88%.

By Chengming Cui, Tianxin Wei, Ziyi Chen, Ruizhong Qiu, Zhichen Zeng, Zhining Liu, Xuying Ning, Duo Zhou, Jingrui He
llmsnlpsafety
More like this →
arXiv Machine Learning
4d ago

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

The paper investigates how large language models (LLMs) used as judges in absolute scoring tasks exhibit systematic biases that compromise reliability. It shows that a judge’s task accuracy strongly predicts both its judging accuracy and its directional bias, yet more capable examinee models consistently receive more lenient judgments. To mitigate these biases, the authors propose a calibrated weighted majority voting (WMV) ensemble that estimates judges’ error rates from inter-judge agreement patterns, achieving near-oracle performance without labeled data and improving both accuracy and fairness.

By Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan, Sulaiman Vesal
llmsbenchmarkssafety
More like this →