← Back to all news
arXiv Machine Learning By Erin George, Deanna Needell, Berk Ustun

Observational Multiplicity

Read the original on arXiv Machine Learning →

The paper introduces the concept of observational multiplicity, where multiple probabilistic classifiers can perform similarly yet produce conflicting predictions, undermining interpretability and safety. It proposes measuring this arbitrariness through a regret metric that captures how predictions could shift with different training labels. The authors present a general method to estimate regret, show it varies across dataset groups, and discuss its use for safety via abstention and targeted data collection.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 3

Simulating Classification Models for Ex-Ante Evaluation of Predict-Then-Optimize Methods

The paper extends ex‑ante evaluation of Predict‑Then‑Optimize methods from binary to multiclass classification by simulating predictions at specified performance levels and mapping prediction errors to decision regret. It introduces a first‑order approximation that estimates regret from individual misclassifications, reducing computational effort. Experiments show the simulation accurately reproduces target performance and that the approximation is close for some problems, though it falters when simultaneous misclassifications interact significantly.

By Pieter Smet
More like this →
arXiv AI
Sep 2

Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity

The paper introduces a new consistency criterion for auditing decision systems that combines ensemble margin with local prediction variability to address predictive multiplicity, or the Rashomon effect. It shows that finite ensembles converge to the expected model’s consistency score as ensemble size and sample count grow, and demonstrates that ensembling models from the Rashomon set reduces unchecked incorrect predictions while keeping diversions moderate. Experiments on transformer and fine‑tuned language models for NLP and tabular classification confirm the method’s effectiveness and stronger alignment with existing multiplicity metrics.

By Sinjini Banerjee, Tim Marrinan, Anand D. Sarwate
llmsfine-tuning
More like this →
arXiv AI
Jun 9

Performative Learning Theory

arXiv:2602. 04402v3 Announce Type: replace-cross Abstract: Performative predictions influence the very outcomes they aim to forecast.

By Julian Rodemann, Unai Fischer-Abaigar, James Bailie, Krikamol Muandet
rag
More like this →