Forecast-Dojo is a replayable environment designed to benchmark and train large language model (LLM) forecasting agents. It integrates resolved prediction‑market questions with dated news, enabling agents to research events and revisit predictions at successive historical dates. The platform includes 1,568 Polymarket events, 18.8 million dated news articles, and supports repeated evaluation, training interactions, and outcome feedback, with evidence that research tools lower Brier scores across 12 tested models, though all models still lag behind historical market forecasts.
By Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang
arXiv:2607. 03015v1 Announce Type: new Abstract: Forecasting future events has attracted growing attention as a testbed for general-purpose AI.
By Yishu Wang, Yuxuan Wang, Jiaqi Deng, Hanyang Tang
arXiv:2606. 11816v1 Announce Type: cross Abstract: Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information.
By Yizhou Chi, Eric Chamoun, Zifeng Ding, Andreas Vlachos
The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.
By Yuxuan Wang, Yishu Wang, Jiaqi Deng
arXiv:2606. 18686v1 Announce Type: new Abstract: Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score.
By Jaeho Lee, Nick Merrill, Ezra Karger
Recent studies on world modeling for Large Language Model (LLM) agents typically formulate the learning objective as next-observation prediction. However, this objective ties supervision to what a transition happens to reveal, which may omit the dynamics most relevant to the agent's current decision.
arXiv:2605.16358v2 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remain...
By Mingtian Tan, Mihir Parmar, Palash Goyal, Chun-Liang Li, Nanyun Peng, Thomas Hartvigsen, Jinsung Yoon, Tomas Pfister
The paper introduces Agent-Authored World Modeling (AAWM), a training method that replaces traditional next-observation prediction with decision-oriented supervision. AAWM constructs training targets by having the agent identify what it needs to know before acting, then retrieves relevant transition evidence across trajectories to synthesize these needs into learning signals. Experiments across multiple environments demonstrate that this decision-aware approach yields more effective learning than conventional next-observation prediction.
By Guangfeng Cai, Kaibing Yang, Shuo He, Yu Li, Shengtian Yang, Jiaqi Lv, Lei Feng
GIFT‑Eval‑CTX is a new benchmark for contextual forecasting that tests whether models can use verified future information to produce accurate time‑series predictions. It builds on GIFT‑Eval and contains 10,000 instances across 20 datasets, each annotated with macro‑level and micro‑level effects derived from actual future events. The benchmark evaluates traditional TSFMs, advanced LLMs, and agentic systems, showing that structured effects can substantially lower MASE for models like GPT‑5.6 Sol and Claude Opus 4.8, with MoiraiAgent achieving the best overall performance.
By Xu Liu, Hanshu Yan, Taha Aksu, Quang Pham, Zhongzheng Qiao, Jun Hao Liew, Silvio Savarese, Doyen Sahoo, Junnan Li
arXiv:2605. 22681v2 Announce Type: replace Abstract: AI systems are increasingly used to support forward-looking scientific judgment, but it remains unclear whether they can form reliable expectations about future scientific advances.
By Sean Wu, Pan Lu, Yupeng Chen, Jonathan Bragg, Yutaro Yamada, Peter Clark, David Clifton, Philip Torr, James Zou, Junchi Yu
The article "Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks" surveys the lack of a standard definition for AI agents and organizes this ambiguity into five dimensions: environmental interaction, learning and adaptation, autonomy, goal‑directed behavior, and temporal coherence. It reviews how each dimension has been conceptualized in prior work and compiles the metrics, benchmarks, and evaluation frameworks used to assess them. The authors also introduce the Agent Compendium, a public digital resource that extends these evaluation methods, aiming to provide a common structure for evaluating and comparing agent capabilities across AI systems.
By Mia Lassiter, Brinnae Bent