arXiv Machine Learning

Hierarchical Utility Calibration for Structured Multiclass Decisions

arXiv Machine Learning
Sep 16

Observational Multiplicity

The paper introduces the concept of observational multiplicity, where multiple probabilistic classifiers can perform similarly yet produce conflicting predictions, undermining interpretability and safety. It proposes measuring this arbitrariness through a regret metric that captures how predictions could shift with different training labels. The authors present a general method to estimate regret, show it varies across dataset groups, and discuss its use for safety via abstention and targeted data collection.

By Erin George, Deanna Needell, Berk Ustun
arXiv Machine Learning
Sep 3

Simulating Classification Models for Ex-Ante Evaluation of Predict-Then-Optimize Methods

The paper extends ex‑ante evaluation of Predict‑Then‑Optimize methods from binary to multiclass classification by simulating predictions at specified performance levels and mapping prediction errors to decision regret. It introduces a first‑order approximation that estimates regret from individual misclassifications, reducing computational effort. Experiments show the simulation accurately reproduces target performance and that the approximation is close for some problems, though it falters when simultaneous misclassifications interact significantly.

By Pieter Smet