arXiv Computation and Language November 5, 2026 By Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang, Zhuohan Xie, Anke Xu, Wenbo Cao, Jiarui Chen, Zehan Li, Jimin Huang, Alejandro Lopez-Lira, Yuehua Tang, Sophia Ananiadou, Jian-Yun Nie

LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

Read the original on arXiv Computation and Language →

The paper introduces FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, designed to test large language models’ (LLMs) temporal self‑knowledge in financial backtesting. Each fact is paired with four prompts that probe the model’s recall, prediction, and self‑reported knowledge, with controls ensuring chance accuracy near 0.27. Results reveal that most frontier models understate their knowledge, with GPT‑5 correctly answering 76% of facts yet claiming to know only 38%, and that declared dates largely eliminate prior‑knowledge claims without affecting accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Sep 1

Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models

arXiv:2605.24564v2 Announce Type: replace Abstract: Backtesting large language models (LLMs) on historical financial data is unreliable when their pre-training data include the evaluated events. An L...

By Weixian Waylon Li, Mengyu Wang, Tiejun Ma

llms benchmarks safety

arXiv AI

Aug 18

Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot

arXiv:2608. 16795v1 Announce Type: cross Abstract: Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable.

By Hui Mao

llms

arXiv Computation and Language

2d ago

FinForesight: A Diagnostic Backtesting Benchmark for Deep Research Systems in Financial Forecasting

FinForesight is a diagnostic backtesting benchmark designed to evaluate deep research systems for financial forecasting. It tests four forecasting tasks—valuation, return ranking, event direction, and future financial fundamentals—across Chinese, U.S., and Hong Kong markets, while also including two diagnostic tasks for counterfactual reasoning and financial calculation. The framework limits evidence to what was available at the forecast date and compares pretraining-era, post-cutoff, and live evaluations to assess how well controlled backtests predict future performance.

By Fengyuan Liu, Mukai Li, Jieshun You, Yilin Zheng, Chenjia Min, Yizhou Zhang, Harper Chen, Qi Liu

benchmarks

Hugging Face Trending Papers

Jul 27

FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting

Large language models (LLMs) can synthesize financial narratives but may express high confidence when evidence is sparse, stale, or contradictory. This failure is especially consequential in forecasting, where filings, news, prices, volume, and technical signals can disagree.

llms rag agents multimodal

arXiv Computation and Language

Sep 30

Can Language Models Learn to Forecast Stock Prices

arXiv:2609.36914v1 Announce Type: new Abstract: Post-training has been shown to significantly improve language models' performance on tasks with verifiable outcomes, including mathematical reasoning,...

By Jiacheng Guo, Suozhi Huang, Shuzhen Li, Yunlong Gao, Zerui Cheng, Jason Ge, Shushu Liang, Zihao Li, Hao Lu, Ming Yin, Shilong Liu, Jiashuo Liu, Xu Kuang, Mengdi Wang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 25

MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarios

arXiv:2606. 24950v1 Announce Type: new Abstract: Financial decision-making is contextual: forecasting prices, valuing companies, and assessing event exposure weigh price history, accounting fundamentals, macroeconomic regime, and contemporaneous text.

By Patara Trirat, Jin Myung Kwak, Jay Heo, Heejun Lee, Sung Ju Hwang

llms fine-tuning benchmarks