The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.
By Yuxuan Wang, Yishu Wang, Jiaqi Deng
arXiv:2609.24862v1 Announce Type: new
Abstract: Agentic time series forecasting concerns systems whose underlying mechanisms evolve, making the relative effectiveness of numerical models, reasoning s...
By Yifan Hu, Xilin Dai, Zhiyuan Qu, Yiding Liu, Zewei Dong, Jiang-ming Yang, Qiang Xu
The paper presents a method to produce logically consistent and accurate forecasts from large language models. By finetuning a supervisory language model that uses reasoning to reconcile inconsistencies, the authors improve the Brier score of a GPT‑5.1 forecaster by 9.3% and reduce structural inconsistency by 82‑90% on ForecastBench. Extensive baselines and ablations show that no other approach achieves comparable gains in both consistency and accuracy.
By Kate Yuan, Rohan Alur, Manish Raghavan, Devavrat Shah
CastWorld is a new benchmark for agentic, context-aware time series forecasting that expands beyond single-shot predictions to include autonomous, interactive forecasting. It offers a context-rich dataset suite covering eight domains and 25 real-world datasets, a modular harness with 49 tools and a state-adaptive loop for iterative evidence gathering and forecast revision, and comprehensive evaluation metrics at both model and system levels. Experiments show that additional interactions do not always improve forecasts, but allowing evidence acquisition during refinement boosts task-aware forecast scores across evaluated models.
By Bokai Pan, Ruijie Yang, Mingyue Cheng, Jiahao Wang, Xiaoyu Tao
Agentic time series forecasting concerns systems whose underlying mechanisms evolve, making the relative effectiveness of numerical models, reasoning strategies, and intervention rules inherently time...
arXiv:2606. 11816v1 Announce Type: cross Abstract: Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information.
By Yizhou Chi, Eric Chamoun, Zifeng Ding, Andreas Vlachos
arXiv:2610. 38225v1 Announce Type: new Abstract: Real-world forecasting often requires integrating heterogeneous evidence whose predictive value varies across horizons, operating regimes, and domains.
By Madumita Karthikeyan, Umang Rajnikant Bid, Rong Pan
arXiv:2604. 18576v4 Announce Type: replace Abstract: We present the Bayesian Linguistic Forecaster (BLF), an agentic system for binary forecasting that achieves state-of-the-art performance on the ForecastBench benchmark.
By Kevin Murphy
arXiv:2609.05905v1 Announce Type: cross
Abstract: LLM agents are increasingly used for live forecasting, where they retrieve up-to-date information and produce estimates for unresolved future events....
By Yuanpu Cao, Yongkang Du, Yurui Chang, Lu Lin, Jinghui Chen
arXiv:2606. 02497v1 Announce Type: new Abstract: Time series forecasting has advanced rapidly, especially with the emergence of foundation models that show strong zero-shot performance on numerical extrapolation.
By Yuhua Liao, Zetian Wang, Qiangqiang Nie, Zhenhua Zhang
The paper introduces Speculative Uncertainty (SU), a technique that infers a failure likelihood for black‑box LLM agents by evaluating their generated token sequences with a lightweight draft model, without needing internal model details. SU extracts phase‑aware features from reasoning and action spans, calibrates them against verifiable outcomes, and produces a failure‑likelihood score usable by downstream policies. Applying a pre‑execution veto gate based on SU to software‑engineering agents such as Qwen3‑Coder‑480B and Claude 3.5 Sonnet reduced execution error rates by 6‑8 percentage points and token costs by 14‑19 %, while maintaining performance on out‑of‑distribution benchmarks and across different agent models.
By Konstantin Grotov, Valentin Malykh
arXiv:2607. 16229v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as components of agentic systems that observe, plan, and act.
By Rishab Ghosh, Vinay Devarakonda