← Back to all news
arXiv Statistics ML By Pin Ni, Francesca Medda, Ramin Okhrati

When a High Score Is an Illusion: Certifying Genuine versus Repackaged Forecasting Skill

Read the original on arXiv Statistics ML →

The paper investigates how reusing observations for ranking forecasts can create artificial associations between forecast and outcome ranks. It develops theoretical conditions that preserve these associations and introduces unbiased kernel and U‑statistic estimators for interaction terms. Empirical results on a Beijing air‑quality archive show that interaction explains over 90% of shared forecast scores, and a matched null experiment demonstrates that distinct references dramatically reduce false rejections.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Statistics ML.

Related stories

arXiv Machine Learning
Sep 24

Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel

The paper demonstrates that the outcome of a forecasting leaderboard is largely determined by the evaluator’s design choices rather than the models themselves. By fixing the data, horizon, and period, the authors varied three key evaluation decisions—unit of analysis, error pooling, and scoring metric—and showed that each can reverse or eliminate the apparent superiority of any forecasting method. The study also evaluates the practical impact of these choices on a deployed system, revealing that the selection rule captures a significant portion of the potential performance gain, and confirms the findings on an external public dataset.

By Md Rezwanul Islam, Wael Mohammed
benchmarks
More like this →
arXiv AI
Sep 4

RATL: Learning from Retrieved Residuals for Robust Multivariate Time-Series Forecasting

RATL is a plug‑in method for multivariate time‑series forecasting that uses a frozen base forecaster to build a memory of its historical forecast residuals. During inference, RATL retrieves residual trajectories from similar past contexts and employs a set‑aware router to combine them, providing learned feedback correction. Experiments demonstrate that this residual‑retrieval approach improves the performance of the base forecaster across various benchmarks and backbones.

By Yuchen He, Yueyang Cang, Zhiyuan Ning, Ningyu Wang, Li Shi
ragbenchmarks
More like this →
arXiv Computation and Language
Aug 27

Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking

The paper demonstrates that open‑ended Theory‑of‑Mind trackers can produce valid beliefs that are absent from finite reference sets, and that treating unmatched outputs as false can reverse model‑selection rankings. By recoding references for 259 beliefs, the authors show a dramatic drop in weighted prevalence and a reversal of strictly proper Brier risk, with similar distortions observed in a 301‑question NQ‑open DPR‑BERT pipeline. The study further reveals that 90‑96% of audited unmatched beliefs are literally true, and introduces a TriSource‑Restore method that anchors reference labels to a probability‑sampled human pilot to restore calibration and ranking integrity.

By Zhexi Feng, Wuxi Chen, Bingrui Zhang
More like this →