arXiv:2607. 17765v1 Announce Type: cross Abstract: We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events.
The paper demonstrates that the outcome of a forecasting leaderboard is largely determined by the evaluator’s design choices rather than the models themselves. By fixing the data, horizon, and period, the authors varied three key evaluation decisions—unit of analysis, error pooling, and scoring metric—and showed that each can reverse or eliminate the apparent superiority of any forecasting method. The study also evaluates the practical impact of these choices on a deployed system, revealing that the selection rule captures a significant portion of the potential performance gain, and confirms the findings on an external public dataset.
Forecast-Dojo is a replayable environment designed to benchmark and train large language model (LLM) forecasting agents. It integrates resolved prediction‑market questions with dated news, enabling agents to research events and revisit predictions at successive historical dates. The platform includes 1,568 Polymarket events, 18.8 million dated news articles, and supports repeated evaluation, training interactions, and outcome feedback, with evidence that research tools lower Brier scores across 12 tested models, though all models still lag behind historical market forecasts.
arXiv:2509. 10600v5 Announce Type: replace-cross Abstract: Collegiate cross country teams often build their season schedules on intuition rather than evidence, partly because large-scale performance datasets were not publicly accessible prior to the National Running Club Database (NRCD).
LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting) is a new approach that reorganizes how evidence is used in LLM-based forecasting systems. Instead of a monolithic prediction that aggregates all evidence at once, LEAP examines each evidence item separately, elicits likelihood parameters, and combines them with an explicit prior to produce a posterior distribution. The method supports continuous, single-choice, and multi-choice forecasts and has been shown to improve prediction and calibration metrics across models on a benchmark covering forecasting, information-seeking, and browsing tasks.
arXiv:2607. 03015v1 Announce Type: new Abstract: Forecasting future events has attracted growing attention as a testbed for general-purpose AI.
arXiv:2401. 14483v4 Announce Type: replace Abstract: In the current practices of machine learning, the evaluation of forecasts has become a cornerstone of scientific progress.
Hedge-Bench is a benchmark comprising 92 real-world environments derived from hedge fund analysts' reasoning traces, designed to test AI agents’ ability to forecast future events from incomplete information. Agents are evaluated on joint success—correct forecasts with over 60% research coverage—across 11 frontier models, achieving rates between 2.54% and 31.52%. Training Muse‑Glimmer with GRPO on 165 environments improves coverage on held‑out tasks and demonstrates generalization to out‑of‑distribution evaluations.
arXiv:2609.36914v1 Announce Type: new Abstract: Post-training has been shown to significantly improve language models' performance on tasks with verifiable outcomes, including mathematical reasoning,...
The study evaluates a multi‑agent large language model system for forecasting outcomes of the 2026 FIFA World Cup. Two specialist agents—one quantitative and one news‑focused—produce forecasts that are then reviewed by a critic and combined by a meta‑agent. Results show the news specialist performs best, matching betting market accuracy, while the meta‑agent adds little beyond the specialists’ predictions.
The paper proposes a method for controlling downside risk when adjusting forecasts from frozen models, such as foundation models, by combining a static corrector and an online corrector on the simplex. Using only post‑horizon losses, the approach achieves minimal deterioration (0.15%) and up to 11.5% gains across 28 forecast pairs, and consistently reduces mean MSE in day‑ahead load forecasts for seven European bidding zones. The method’s applicability is bounded by three empirical conditions related to expert speed, stream length, and outcome alignment.
Vendster is an evaluation arena that studies how the bias and prediction‑interval width of time‑series forecasters influence the outcomes of buyer‑vendor discount negotiations conducted by large language model (LLM) agents. In its reference instance, Vendster‑M5, forecasts generated by LLMs increased sales target attainment by up to 17 percentage points, yet traditional forecast accuracy metrics did not predict negotiation success. Controlled experiments revealed that under‑prediction bias significantly improves outcomes for agents that react strongly to forecasts, while interval width has no notable effect when bias is zero.