arXiv AI

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Forecast-Dojo is a replayable environment designed to benchmark and train large language model (LLM) forecasting agents. It integrates resolved prediction‑market questions with dated news, enabling agents to research events and revisit predictions at successive historical dates. The platform includes 1,568 Polymarket events, 18.8 million dated news articles, and supports repeated evaluation, training interactions, and outcome feedback, with evidence that research tools lower Brier scores across 12 tested models, though all models still lag behind historical market forecasts.

arXiv AI
2d ago

CastWorld: Benchmarking Context-Aware Agentic Time Series Forecasting Beyond Predictive Accuracy

CastWorld is a new benchmark for agentic, context-aware time series forecasting that expands beyond single-shot predictions to include autonomous, interactive forecasting. It offers a context-rich dataset suite covering eight domains and 25 real-world datasets, a modular harness with 49 tools and a state-adaptive loop for iterative evidence gathering and forecast revision, and comprehensive evaluation metrics at both model and system levels. Experiments show that additional interactions do not always improve forecasts, but allowing evidence acquisition during refinement boosts task-aware forecast scores across evaluated models.

By Bokai Pan, Ruijie Yang, Mingyue Cheng, Jiahao Wang, Xiaoyu Tao
arXiv Machine Learning
Nov 2

Benchmarking Forecasting Agents on Yesterday's Tomorrows

The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.

By Yuxuan Wang, Yishu Wang, Jiaqi Deng
arXiv AI
Oct 19

CastFlow: Bridging Evidence Acquisition and Accurate Prediction in Agentic Time Series Forecasting

CastFlow is a role‑specialized agentic forecasting framework that combines general‑purpose tool reasoning with specialized temporal forecasting. It uses an Experience Memory to guide a frozen LLM in acquiring evidence via a multi‑view toolkit, then a lightweight specialized LLM refines predictions through evidence‑grounded reasoning. The two‑stage training strategy—supervised fine‑tuning followed by reinforcement learning with verifiable rewards—optimizes the specialized model based on final forecasting outcomes, achieving superior performance on nine out of ten real‑world benchmarks.

By Bokai Pan, Mingyue Cheng, Xiaoyu Tao, Zhiding Liu, Shuo Yu, Yuchong Wu, Qi Liu, Defu Lian, Enhong Chen
arXiv AI
Aug 25

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

arXiv:2608.23058v1 Announce Type: new Abstract: Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external too...

By Xiaogang Xu, Jiaqi Tang, Jianmin Chen, Yingying Yan, Zhenchao Tang, Xiangxin Zhou, Xiaobin Hu, Wei Wei, Jinfeng Wu, Qifeng Chen, Lu Zhou, Jiafei Wu, Zhe Liu, Jianwei Yin, Weimin Zheng
arXiv Machine Learning
Oct 3

Do Your Own Research: Learning to Forecast by Learning to Search

The paper introduces an agentic forecasting environment built on 2,100+ resolved Polymarket questions, where a language model (Qwen3.5-35B-A3B) learns to gather evidence during rollout via web search, page reading, and financial time series, all filtered to avoid post‑cutoff leaks. Training with single‑epoch GRPO and a Brier‑score reward improves calibration by 30‑40% and reduces search attempts, while the trained policy outperforms four frontier models in evidence‑based forecasting, achieving lower soft‑Brier scores at roughly 5% of the inference cost. The authors release the environment, dataset, and per‑rollout records as a reusable harness for temporal forecasting agents.

By Yusuf Afifi, Artur Kiulian, Anton Polishko, Mykola Khandoga, Hamudi Naanaa, Alina Krasnobrizha
arXiv AI
23h ago

Logically Consistent Accurate Forecasts

The paper presents a method to produce logically consistent and accurate forecasts from large language models. By finetuning a supervisory language model that uses reasoning to reconcile inconsistencies, the authors improve the Brier score of a GPT‑5.1 forecaster by 9.3% and reduce structural inconsistency by 82‑90% on ForecastBench. Extensive baselines and ablations show that no other approach achieves comparable gains in both consistency and accuracy.

By Kate Yuan, Rohan Alur, Manish Raghavan, Devavrat Shah
arXiv AI
Oct 19

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

AlphaBet is a reinforcement learning framework that trains a single large language model to search for evidence, reason about it, and report probabilities for unresolved events across multiple pre-resolution dates. By scoring each dated forecast with a proper scoring rule, AlphaBet uses one resolved event to supervise decisions under varying information states, decomposing forecasting regret into evidence-acquisition and probability-reporting gaps. Experiments on GLM-4.5-Air show that AlphaBet improves Brier scores on Forecast-Dojo and FutureX, attributing gains to multi-date training and better information acquisition and probability reporting.

By Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang
arXiv Computation and Language
Sep 30

Can Language Models Learn to Forecast Stock Prices

arXiv:2609.36914v1 Announce Type: new Abstract: Post-training has been shown to significantly improve language models' performance on tasks with verifiable outcomes, including mathematical reasoning,...

By Jiacheng Guo, Suozhi Huang, Shuzhen Li, Yunlong Gao, Zerui Cheng, Jason Ge, Shushu Liang, Zihao Li, Hao Lu, Ming Yin, Shilong Liu, Jiashuo Liu, Xu Kuang, Mengdi Wang