arXiv AI

When Should Forecasting Agents Reason? Behavioral Stress Tests for Reliability Routing

The paper investigates when forecasting agents should employ different behaviors—retrieval, reasoning, deferring to market priors, or using historical analogs—on binary forecasting tasks. It finds that the optimal mechanism depends on the data source, with structured analogs excelling for some processes and market or conservative baselines for others. The authors propose ReliabilityRoute, a rule‑based system that steers agent behavior using reliability features, achieving competitive performance across multiple LLM versions while highlighting that more reasoning is not always better.

arXiv Machine Learning
Nov 2

Benchmarking Forecasting Agents on Yesterday's Tomorrows

The paper introduces FABench, a scalable backtesting environment designed to evaluate forecasting agents. FABench automatically discovers events, generates questions, and resolves outcomes for any past forecast date, allowing continuous renewal beyond model training cutoffs. In initial tests, retrieval improves Brier scores across eight models, with top agents matching or surpassing market consensus on 54 Polymarket-linked questions, yet analysis shows persistent evidence‑weighting failures that highlight room for improvement.

By Yuxuan Wang, Yishu Wang, Jiaqi Deng
arXiv AI
22h ago

Logically Consistent Accurate Forecasts

The paper presents a method to produce logically consistent and accurate forecasts from large language models. By finetuning a supervisory language model that uses reasoning to reconcile inconsistencies, the authors improve the Brier score of a GPT‑5.1 forecaster by 9.3% and reduce structural inconsistency by 82‑90% on ForecastBench. Extensive baselines and ablations show that no other approach achieves comparable gains in both consistency and accuracy.

By Kate Yuan, Rohan Alur, Manish Raghavan, Devavrat Shah
arXiv AI
2d ago

CastWorld: Benchmarking Context-Aware Agentic Time Series Forecasting Beyond Predictive Accuracy

CastWorld is a new benchmark for agentic, context-aware time series forecasting that expands beyond single-shot predictions to include autonomous, interactive forecasting. It offers a context-rich dataset suite covering eight domains and 25 real-world datasets, a modular harness with 49 tools and a state-adaptive loop for iterative evidence gathering and forecast revision, and comprehensive evaluation metrics at both model and system levels. Experiments show that additional interactions do not always improve forecasts, but allowing evidence acquisition during refinement boosts task-aware forecast scores across evaluated models.

By Bokai Pan, Ruijie Yang, Mingyue Cheng, Jiahao Wang, Xiaoyu Tao
arXiv Machine Learning
Sep 7

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method

The paper introduces Speculative Uncertainty (SU), a technique that infers a failure likelihood for black‑box LLM agents by evaluating their generated token sequences with a lightweight draft model, without needing internal model details. SU extracts phase‑aware features from reasoning and action spans, calibrates them against verifiable outcomes, and produces a failure‑likelihood score usable by downstream policies. Applying a pre‑execution veto gate based on SU to software‑engineering agents such as Qwen3‑Coder‑480B and Claude 3.5 Sonnet reduced execution error rates by 6‑8 percentage points and token costs by 14‑19 %, while maintaining performance on out‑of‑distribution benchmarks and across different agent models.

By Konstantin Grotov, Valentin Malykh