← Back to all news
arXiv Machine Learning

The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark

The paper audits the IBM Telco Customer Churn benchmark, revealing that common practices inflate performance metrics. It shows that pre‑split SMOTE boosts churn‑class F1 by 13.1 points, that isotonic regression is the best calibration method while temperature scaling fails on tree ensembles, and that the cost‑optimal decision threshold is 5–10 times lower than the F1‑optimal one, saving about $77,000 per 1,000 customers. The authors also test generalisation on Iranian Telecom and Bank churn datasets, and propose a four‑component reporting checklist with reproducible code.

Related stories

arXiv AI
Aug 28

Explainable Artificial Intelligence for Customer Churn Prediction in Telecommunications: A Framework for CRM Integration

The paper presents a framework for integrating explainable AI into customer churn prediction for telecommunications. It benchmarks four classifiers—Logistic Regression, Random Forest, XGBoost, and LightGBM—on the IBM Telco Customer Churn dataset, finding comparable performance with Logistic Regression achieving the highest AUC-ROC and LightGBM the highest accuracy. Explanations are provided via SHAP and LIME at both global and instance levels, and a four‑layer CRM integration architecture is proposed to translate risk scores and attribution vectors into actionable retention strategies, projecting a 3.3–5.3 percentage point reduction in churn and $199K–$319K savings per campaign cycle.

By Sandeep Gaddamwar
computer-visionbenchmarks
More like this →
arXiv Machine Learning
Jul 30

Cost-Sensitive Conformal Prediction and Human-in-the-Loop Abstention for Imbalanced High-Stakes Decision Support: A Multi-Domain Benchmark

arXiv:2607. 27143v1 Announce Type: new Abstract: High-stakes decision systems in credit scoring, fraud detection, healthcare, and industrial safety require reliable uncertainty quantification under severe class imbalance and asymmetric error costs.

By Manpreet Singh, Akshatha Srikantha, Shyamal Lakhanpal
benchmarkssafety
More like this →
arXiv Machine Learning
Sep 24

CRISP: Scalable Importance-Stratified Coresets for Imbalanced Tabular Learning

CRISP (Coreset Reduction via Importance-Stratified Pruning) is a linear-time method that reduces negative-class examples in highly imbalanced tabular datasets by allocating a budget across quantile strata of a proxy-model score and using sample weights to correct for unequal inclusion probabilities. On a production fraud dataset, CRISP cuts the training set from 25 M to about 1.70 M rows (a 93.2% reduction) while preserving 99.7% of the full-data Average Precision. In public benchmarks such as CriteoPrivateAds, CRISP consistently achieves the highest mean Average Precision across a range of majority reductions, with ablation studies highlighting budget allocation and inverse-propensity weighting as key contributors to its performance.

By Hardhik Mohanty, Indrayana Rustandi, Mohamadreza Sheibani
efficiency
More like this →