← Back to all news
arXiv AI By Joran Leest, Ilias Gerostathopoulos, Claudia Raibulet, Patricia Lago

Counterfactual Model Testing Under Distribution Shift

Read the original on arXiv AI →

The paper addresses the problem of evaluating deployed machine learning models when their input data distribution shifts, which can arise from changes in the underlying population or from data integrity issues in the feature pipeline. It proposes an online model‑testing framework that reproduces the detected shift on a controlled baseline with known labels, distinguishing between the two mechanisms of shift to estimate current performance and potential gains from repairing data integrity problems. Experiments on real‑world datasets with simulated faults demonstrate that separating the two sources of shift generally improves performance estimates, and a churn‑prediction case study shows how these tests can guide monitoring and repair prioritization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 28

Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

The paper introduces a unified evaluation protocol for robust counterfactual explanations (CFE), testing six robust methods and two baselines across four tabular datasets under eight types of model change. It shows that robustness scores vary by change type and that methods designed for one change family may not transfer to others, with RobX performing most consistently. The study emphasizes the need for a common protocol that defines model changes, measures their impact, and separates generation performance from robustness.

By Marcin Kostrzewa, Maciej Zi\k{e}ba
benchmarks
More like this →
arXiv AI
6d ago

Counterfactual Test-Time Augmentation for Shortcut Removal in Language Models

The paper introduces Counterfactual Test-Time Augmentation (CTTA) to mitigate shortcut exploitation in large language models. CTTA generates counterfactual edits of an input using an auxiliary model, then averages the LLM’s predictions across these edits, with theoretically optimal weights derived from the training distribution. The authors propose Select-CTTA, which selects among CTTA variants based on tests for assumption violations, and demonstrate that it outperforms other test-time methods and rivals models fine‑tuned on shortcut‑balanced data.

By Katie Matton, Rosalind W. Picard, John Guttag, Maggie Makar
llmsfine-tuning
More like this →