arXiv Machine Learning

Benchmarking Sensor-Fault Robustness in Forecasting

arXiv AI
Sep 17

Reliable Virtual Sensing: A Multi-Domain Benchmark for Robustness Under Sensor Failures

The paper introduces MuViS-C, a multi‑domain benchmark that evaluates the robustness of learning‑based virtual sensing models against ten common sensor failure modes, ranging from subtle drifts to catastrophic dropouts. It assesses models using average error, relative degradation, and worst‑case fragility across nine datasets from six domains, comparing six architectures (gradient‑boosted trees, convolution, recurrence, attention, and MLP‑mixing). The study finds that all models degrade under corruption, gradient‑boosted trees are most robust, and targeted robustification can improve attention models at the cost of nominal performance.

By Jens U. Brandt, Noah C. Puetz, Alexander Windmann, Marc Hilbert, Elena Raponi, Thomas B\"ack, Thomas Bartz-Beielstein
arXiv AI
Oct 5

Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

The paper argues that current time‑series forecasting benchmarks are too narrow, rewarding low held‑out error while failing to capture real‑world failure modes such as structured input anomalies, regime shifts, and sensor faults. It proposes scenario‑grounded stress testing, where each test includes historical data, future targets, a semantic scenario, a failure operator, and a difficulty level, to make evaluation interpretable and deployment‑relevant.

By Yuyang Zhao, Lian Xu, Hao Xue
arXiv Machine Learning
Oct 5

Validated Data Onboarding for AI Demand Forecasting on U.S. Building Meter Data: Design, Controlled Evaluation, and a Corrected Negative Result

The paper introduces a data‑onboarding pipeline that detects and repairs defects in building meter data before training machine‑learning demand‑forecasting models. Controlled experiments on hourly electricity data from twelve U.S. buildings show that the pipeline restores forecast accuracy to clean‑data levels, even when defects affect up to 1.6% of training rows, and works across multiple models and horizons. The study also documents a first‑version over‑cleaning issue, its cause, and a per‑building calibration that corrects it, with all results reproducible from publicly pinned inputs.

By Yixuan Liang
arXiv AI
Oct 27

EvoBench: An Evolving, Diagnostic Benchmark Framework for Environmental Modeling

EvoBench is a diagnostic benchmark framework designed to reveal where and when machine‑learning models for environmental variables fail, rather than relying on a single average accuracy score. It tracks model performance over time and across spatial extrapolation, highlighting issues such as non‑stationarity, poor performance at unmonitored sites, and under‑prediction of rare events like floods. In a case study on daily streamflow prediction across 9,000 U.S. basins, EvoBench shows that no model dominates all scenarios and that targeted fixes can mitigate identified weaknesses.

By Yiming Sun, Yuan Zhang, Qi Cheng, Yingda Fan, Runlong Yu, Yiqun Xie, Xiaowei Jia