arXiv AI
OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 5
LEAF: A Living Benchmark for Event-Augmented Forecasting
arXiv:2605.16358v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remain...
arXiv AI
Aug 25
LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
arXiv:2608.23058v1 Announce Type: new Abstract: Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external too...
arXiv AI
Oct 19
CastFlow: Bridging Evidence Acquisition and Accurate Prediction in Agentic Time Series Forecasting
CastFlow is a role‑specialized agentic forecasting framework that combines general‑purpose tool reasoning with specialized temporal forecasting. It uses an Experience Memory to guide a frozen LLM in acquiring evidence via a multi‑view toolkit, then a lightweight specialized LLM refines predictions through evidence‑grounded reasoning. The two‑stage training strategy—supervised fine‑tuning followed by reinforcement learning with verifiable rewards—optimizes the specialized model based on final forecasting outcomes, achieving superior performance on nine out of ten real‑world benchmarks.
arXiv Machine Learning
Jun 4
It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks
arXiv:2602. 12147v4 Announce Type: replace Abstract: Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.
arXiv AI
Jun 18
SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents
arXiv:2606. 18733v1 Announce Type: cross Abstract: Realistic coding-agent benchmarks often replay public GitHub issues and pull requests, making them vulnerable to overlap with model pretraining, fine-tuning, synthetic-data generation, or benchmark-driven model selection.
arXiv Machine Learning
Jul 21
Efficient Sequential Evaluation of Large Language Models
arXiv:2607. 17409v1 Announce Type: cross Abstract: We study the problem of sequentially evaluating a new large language model (LLM) on a fixed question set using historical performance data from prior LLMs.