arXiv AI

Logically Consistent Accurate Forecasts

The paper presents a method to produce logically consistent and accurate forecasts from large language models. By finetuning a supervisory language model that uses reasoning to reconcile inconsistencies, the authors improve the Brier score of a GPT‑5.1 forecaster by 9.3% and reduce structural inconsistency by 82‑90% on ForecastBench. Extensive baselines and ablations show that no other approach achieves comparable gains in both consistency and accuracy.

arXiv AI
Aug 25

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

arXiv:2608.23058v1 Announce Type: new Abstract: Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external too...

By Xiaogang Xu, Jiaqi Tang, Jianmin Chen, Yingying Yan, Zhenchao Tang, Xiangxin Zhou, Xiaobin Hu, Wei Wei, Jinfeng Wu, Qifeng Chen, Lu Zhou, Jiafei Wu, Zhe Liu, Jianwei Yin, Weimin Zheng
arXiv AI
2d ago

CastWorld: Benchmarking Context-Aware Agentic Time Series Forecasting Beyond Predictive Accuracy

CastWorld is a new benchmark for agentic, context-aware time series forecasting that expands beyond single-shot predictions to include autonomous, interactive forecasting. It offers a context-rich dataset suite covering eight domains and 25 real-world datasets, a modular harness with 49 tools and a state-adaptive loop for iterative evidence gathering and forecast revision, and comprehensive evaluation metrics at both model and system levels. Experiments show that additional interactions do not always improve forecasts, but allowing evidence acquisition during refinement boosts task-aware forecast scores across evaluated models.

By Bokai Pan, Ruijie Yang, Mingyue Cheng, Jiahao Wang, Xiaoyu Tao
arXiv Machine Learning
Nov 2

Benchmarking Forecasting Agents on Yesterday's Tomorrows

The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.

By Yuxuan Wang, Yishu Wang, Jiaqi Deng
arXiv AI
Oct 19

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

AlphaBet is a reinforcement learning framework that trains a single large language model to search for evidence, reason about it, and report probabilities for unresolved events across multiple pre-resolution dates. By scoring each dated forecast with a proper scoring rule, AlphaBet uses one resolved event to supervise decisions under varying information states, decomposing forecasting regret into evidence-acquisition and probability-reporting gaps. Experiments on GLM-4.5-Air show that AlphaBet improves Brier scores on Forecast-Dojo and FutureX, attributing gains to multi-date training and better information acquisition and probability reporting.

By Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang
arXiv AI
Oct 19

CastFlow: Bridging Evidence Acquisition and Accurate Prediction in Agentic Time Series Forecasting

CastFlow is a role‑specialized agentic forecasting framework that combines general‑purpose tool reasoning with specialized temporal forecasting. It uses an Experience Memory to guide a frozen LLM in acquiring evidence via a multi‑view toolkit, then a lightweight specialized LLM refines predictions through evidence‑grounded reasoning. The two‑stage training strategy—supervised fine‑tuning followed by reinforcement learning with verifiable rewards—optimizes the specialized model based on final forecasting outcomes, achieving superior performance on nine out of ten real‑world benchmarks.

By Bokai Pan, Mingyue Cheng, Xiaoyu Tao, Zhiding Liu, Shuo Yu, Yuchong Wu, Qi Liu, Defu Lian, Enhong Chen
arXiv AI
Aug 5

CastFSR: A Fast--Slow--Reflect Agentic Reasoning Framework for Context-Aware Time Series Forecasting

arXiv:2608. 03031v1 Announce Type: new Abstract: Time series forecasting is fundamental to decision-making in complex systems, where future dynamics are influenced not only by historical observations but also by evolving contextual features.

By Xiaoyu Tao, Mingyue Cheng, Bokai Pan, Chuang Jiang, Huanjian Zhang, Tian Gao, Yaguo Liu, Qi Liu, Enhong Chen
arXiv AI
Sep 25

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Forecast-Dojo is a replayable environment designed to benchmark and train large language model (LLM) forecasting agents. It integrates resolved prediction‑market questions with dated news, enabling agents to research events and revisit predictions at successive historical dates. The platform includes 1,568 Polymarket events, 18.8 million dated news articles, and supports repeated evaluation, training interactions, and outcome feedback, with evidence that research tools lower Brier scores across 12 tested models, though all models still lag behind historical market forecasts.

By Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang