EPSBench is a benchmark that evaluates large language model agents on quarterly earnings forecasting by having them connect financial evidence to income‑statement line items for 149 firm‑quarters from 86 US companies. Agents read filings, earnings‑call transcripts, and news, estimate company‑specific drivers, and a shared accounting calculator converts these into GAAP diluted earnings per share (EPS). Six models were assessed, with three showing no significant difference from analyst consensus in point forecast error, while the others performed worse; nominal 80% prediction intervals contained reported EPS only 37–70% of the time, largely due to low operating‑income estimates.
arXiv:2608. 01599v1 Announce Type: new Abstract: Volatility forecasts are commonly evaluated with aggregate accuracy metrics such as RMSE and MAE, but these metrics can hide conditional failures that matter for risk management.
The study investigates which inputs influence a language model’s financial forecasts by creating synthetic earnings announcements that independently reverse 20‑day momentum, commentary tone, and EPS surprise. Six instruction/chat checkpoint conversions from four model families (0.49‑7.61 B parameters) were tested with 2,016 prompts each, covering 384 primary scenarios. Results show that the contrast in primary EPS ranges from –0.19 to 86.80 percentage points, with signed tone and momentum contrasts often exceeding EPS effects; numerical diagnostic accuracy varied from 51.0 % to 95.8 %. The experiment measures observable input reliance and numerical recognition but does not establish internal reasoning mechanisms or return predictability.
arXiv:2512. 23847v2 Announce Type: replace-cross Abstract: We develop a statistical procedure to detect lookahead bias in economic forecasts generated by large language models (LLMs).
arXiv:2609.05905v1 Announce Type: cross Abstract: LLM agents are increasingly used for live forecasting, where they retrieve up-to-date information and produce estimates for unresolved future events....
arXiv:2607. 16229v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as components of agentic systems that observe, plan, and act.
arXiv:2606. 15640v1 Announce Type: new Abstract: Audit risk assessment increasingly benefits from combining heterogeneous evidence sources, yet existing approaches typically produce point predictions without quantifying how well different evidence streams agree.
Specialist training beats generalist scale when forecasting financial statements. To our knowledge, no prior work jointly forecasts complete financial statements beyond one year, yet in a discounted-cash-flow valuation most firm value sits past that window.
arXiv:2608. 11327v1 Announce Type: new Abstract: Specialist training beats generalist scale when forecasting financial statements.
The paper introduces an Availability Contract to address timing mismatches in financial NLP benchmarks, requiring explicit declaration of input boundaries, evidence of availability, outcome start, and supported claims. Applying this framework to 552 earnings calls, the authors show that shifting from a call‑start proxy to a completion proxy changes return signs in 9.4% of calls and alters quintile rankings in 28.3%. In a volatility benchmark, moving target windows forward by one session increases mean squared error by 18.1‑21.4% while leaving point rankings unchanged, illustrating that labels and losses can be sensitive to the declared outcome clock.
Large language models (LLMs) can synthesize financial narratives but may express high confidence when evidence is sparse, stale, or contradictory. This failure is especially consequential in forecasting, where filings, news, prices, volume, and technical signals can disagree.
LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting) is a new approach that reorganizes how evidence is used in LLM-based forecasting systems. Instead of a monolithic prediction that aggregates all evidence at once, LEAP examines each evidence item separately, elicits likelihood parameters, and combines them with an explicit prior to produce a posterior distribution. The method supports continuous, single-choice, and multi-choice forecasts and has been shown to improve prediction and calibration metrics across models on a benchmark covering forecasting, information-seeking, and browsing tasks.
arXiv:2608. 03259v1 Announce Type: cross Abstract: As time-series foundation models have emerged, the need for benchmarks that can evaluate their forecasting ability in meaningful ways has become increasingly important.