The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.
By Yuxuan Wang, Yishu Wang, Jiaqi Deng
arXiv:2609.24862v1 Announce Type: new
Abstract: Agentic time series forecasting concerns systems whose underlying mechanisms evolve, making the relative effectiveness of numerical models, reasoning s...
By Yifan Hu, Xilin Dai, Zhiyuan Qu, Yiding Liu, Zewei Dong, Jiang-ming Yang, Qiang Xu
The paper presents a method to produce logically consistent and accurate forecasts from large language models. By finetuning a supervisory language model that uses reasoning to reconcile inconsistencies, the authors improve the Brier score of a GPT‑5.1 forecaster by 9.3% and reduce structural inconsistency by 82‑90% on ForecastBench. Extensive baselines and ablations show that no other approach achieves comparable gains in both consistency and accuracy.
By Kate Yuan, Rohan Alur, Manish Raghavan, Devavrat Shah
CastWorld is a new benchmark for agentic, context-aware time series forecasting that expands beyond single-shot predictions to include autonomous, interactive forecasting. It offers a context-rich dataset suite covering eight domains and 25 real-world datasets, a modular harness with 49 tools and a state-adaptive loop for iterative evidence gathering and forecast revision, and comprehensive evaluation metrics at both model and system levels. Experiments show that additional interactions do not always improve forecasts, but allowing evidence acquisition during refinement boosts task-aware forecast scores across evaluated models.
By Bokai Pan, Ruijie Yang, Mingyue Cheng, Jiahao Wang, Xiaoyu Tao
Agentic time series forecasting concerns systems whose underlying mechanisms evolve, making the relative effectiveness of numerical models, reasoning strategies, and intervention rules inherently time...
arXiv:2606. 11816v1 Announce Type: cross Abstract: Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information.
By Yizhou Chi, Eric Chamoun, Zifeng Ding, Andreas Vlachos