The paper demonstrates that the outcome of a forecasting leaderboard is largely determined by the evaluator’s design choices rather than the models themselves. By fixing the data, horizon, and period, the authors varied three key evaluation decisions—unit of analysis, error pooling, and scoring metric—and showed that each can reverse or eliminate the apparent superiority of any forecasting method. The study also evaluates the practical impact of these choices on a deployed system, revealing that the selection rule captures a significant portion of the potential performance gain, and confirms the findings on an external public dataset.
By Md Rezwanul Islam, Wael Mohammed
arXiv:2609.13840v1 Announce Type: new
Abstract: A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters ar...
By Joo Ern Chin, Shih-Fen Cheng, Aldy Gunawan
arXiv:2607. 12248v1 Announce Type: cross Abstract: Large pretrained time-series models such as TimesFM are attractive for financial forecasting, but raw directional accuracy is a misleading scoreboard in equity markets.
By Taizhen Cheung, SA Kwon
The paper investigates how different proper scoring rules influence the performance and behavior of large language model (LLM) forecasters. Five scoring rules were compared as training objectives for binary forecasts of real-world events, revealing that while they all theoretically incentivize truthful probability reporting, they produce models with varying calibration, probability usage, and bias, information, and noise profiles. The Brier-trained model achieved the lowest Brier score and highest AUC-ROC, whereas the log-trained model achieved the best log score and lowest calibration error, indicating that scoring rule choice can shape both forecast accuracy and error structure.
By Benjamin Turtel, Paul Wilczewski, Kris Skotheim, Ville A. Satop\"a\"a, Philip E. Tetlock
arXiv:2604. 16238v2 Announce Type: replace Abstract: Decision-makers rely on weather forecasts to plant crops, manage wildfires, allocate water and energy, and prepare for weather extremes.
By Hannah Guan, Soukayna Mouatadid, Paulo Orenstein, Judah Cohen, Haiyu Dong, Zekun Ni, Jeremy Berman, Genevieve Flaspohler, Alex Lu, Jakob Schloer, Joshua Talib, Jonathan A. Weyn, Lester Mackey
arXiv:2608. 10433v1 Announce Type: new Abstract: Forecast accuracy does not tell us which past inputs produced a prediction.
By Qipeng Qian, Yuntao Qian
Forecast accuracy does not tell us which past inputs produced a prediction. We separate three questions for time-series models with known delay structure: can the true delay be recovered from the observed data, does the model report it, and does the forecast actually use the same history?
arXiv:2606. 25068v1 Announce Type: new Abstract: Online time-series forecasters receive labels only after horizon-dependent delays, while every adaptation step spends limited compute.
By Xibai Wang
The paper introduces Probabilistic Bias Correction (PBC), a machine learning framework that learns to correct historical probabilistic forecasts, thereby reducing systematic errors in subseasonal weather predictions. Applied to leading dynamical and AI models from ECMWF, PBC doubles the AI system’s modest subseasonal skill and improves the operationally-debiased dynamical model for most pressure, temperature, and precipitation targets. In ECMWF’s 2025 real‑time forecasting competition, PBC’s global forecasts ranked first across all weather variables and lead times, outperforming multiple operational and ensemble models.
By Hannah Guan, Soukayna Mouatadid, Paulo Orenstein, Judah Cohen, Haiyu Dong, Zekun Ni, Jeremy Berman, Genevieve Flaspohler, Alex Lu, Jakob Schloer, Joshua Talib, Jonathan A. Weyn, Lester Mackey
arXiv:2606. 04342v1 Announce Type: cross Abstract: Multi-step time series forecasting (MSF) is commonly evaluated using point-wise error metrics such as mean squared error (MSE), implicitly treating the conditional mean as a sufficient target.
By Riku Green, Zahraa S. Abdallah, Telmo M Silva Filho
arXiv:2609. 20193v1 Announce Type: new Abstract: Retrieval plug-ins supply a deep forecaster with information its lookback window cannot carry.
By Mert Onur Cakiroglu, Elham Buxton, Mehmet Dalkilic, Hasan Kurban
Dynamic Regime-Aware Conformal Prediction (DRACP) is a new method that blends density‑ratio estimation, localized kernel weighting, and probabilistic regime‑aware weighting with a self‑tuning online significance controller to produce reliable prediction intervals under multiple distribution shifts. The authors prove finite‑sample validity with oracle weights, provide a coverage‑gap bound for estimated weights, and give deterministic or regret guarantees for the online controller. In experiments on 48 real forecasting series—including euro‑area inflation, US macroeconomic and energy indicators, and daily financial data—DRACP achieves the most reliable calibration, maintaining coverage close to the nominal 0.90 and never falling below 0.80, while other methods achieve narrower intervals but with higher under‑coverage.
whyItMatters":"DRACP offers a principled trade‑off between calibration and efficiency, ensuring that prediction intervals meet coverage standards even when economic data exhibit covariate shift, concept drift, and latent regimes."
By Bogdan Oancea