Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations
Read the original on arXiv Machine Learning →The paper evaluates five multivariate two-sample drift detectors on large‑scale e‑commerce data, including a 137.5‑million‑row Trendyol feature table. Distributed Maximum Mean Discrepancy with Random Fourier Features on Apache Spark shows strong performance, achieving a Pearson correlation of 0.940 with expected drift magnitude, an 80.4% true positive rate, and a 3.2% false positive rate. In contrast, the per‑dimension Kolmogorov‑Smirnov test fails due to statistic saturation from ID‑like columns, and detectors struggle under weak drift configurations, underscoring limitations in current sampling designs and sensitivity analysis.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.