← Back to all news
arXiv Machine Learning By Pieter Smet

Simulating Classification Models for Ex-Ante Evaluation of Predict-Then-Optimize Methods

Read the original on arXiv Machine Learning →

The paper extends ex‑ante evaluation of Predict‑Then‑Optimize methods from binary to multiclass classification by simulating predictions at specified performance levels and mapping prediction errors to decision regret. It introduces a first‑order approximation that estimates regret from individual misclassifications, reducing computational effort. Experiments show the simulation accurately reproduces target performance and that the approximation is close for some problems, though it falters when simultaneous misclassifications interact significantly.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 16

Observational Multiplicity

The paper introduces the concept of observational multiplicity, where multiple probabilistic classifiers can perform similarly yet produce conflicting predictions, undermining interpretability and safety. It proposes measuring this arbitrariness through a regret metric that captures how predictions could shift with different training labels. The authors present a general method to estimate regret, show it varies across dataset groups, and discuss its use for safety via abstention and targeted data collection.

By Erin George, Deanna Needell, Berk Ustun
safety
More like this →
arXiv Statistics ML
Oct 26

Contextual Decision Making: Predictive or Generative

The paper compares two strategies for contextual decision making: predict‑then‑optimize (PtO), which learns the conditional expected objective, and generate‑then‑optimize (GtO), which learns conditional uncertainty and optimizes over generated scenarios. It analyzes how the choice of learning target influences statistical accuracy and optimization tractability, showing that smoothness can compensate for GtO’s higher dimensionality. The study also discusses tradeoffs in convexity preservation, scenario‑wise convexity, and model flexibility, supported by synthetic experiments and a real case study.

By L. Jeff Hong, Weihuan Huang, Xinyao Li
More like this →