The paper investigates how large language models (LLMs) can produce incorrect answers to financial queries that are underspecified, a phenomenon termed "frame collapse." By testing models on Indian mutual fund data—where product variants and tax rules are formally defined—the authors show that LLMs often rely on ambiguous labels, default assumptions, or outdated rules, leading to answers that match the data but answer a different question. The study demonstrates that even a judge that checks for mislabelled figures can miss these errors if the model silently chooses an alternate interpretation.
arXiv:2604. 24668v3 Announce Type: replace Abstract: Given the increased use of LLMs in financial systems today, it becomes important to evaluate the safety and robustness of such systems.
arXiv:2609.24002v1 Announce Type: new Abstract: Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specifi...
arXiv:2607.11414v2 Announce Type: replace Abstract: Large language models (LLMs) in financial applications fail most consequentially when they are confidently wrong. Hedged, uncertain answers invite...
arXiv:2608. 13706v1 Announce Type: cross Abstract: Existing defenses against hallucination in retrieval-augmented and multi-agent pipelines remain partial: evidence is trusted despite modality disagreement, debate verifies an aggregate report rather than individual claims, and such verification occurs only after drafting, leaving inter-agent errors undetected until the final text.
AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.
arXiv:2607. 20645v1 Announce Type: cross Abstract: We introduce Frontier Financial Judgement, a challenging new benchmark developed in collaboration with professional equity analysts to assess agents' ability to replicate expert human judgements.
The study examines how large language models (LLMs) provide financial advice based on a single risk‑tolerance question, finding that both human respondents and LLMs exhibit significant variability when the same question is repeated. In a national US panel, 41% of households changed their risk answer after two years, and in a lab retest a third changed by two or more points on a 0‑10 scale after six months. When LLMs were given 800 real household profiles with only the risk answer altered, the recommended stock allocation shifted by 10–16 percentage points on average, and by 24–35 points when the answer changed, compared to minimal swings when the same prompt was repeated. The study also shows that adding clarifying questions or noting answer noise can reduce these swings, especially for younger clients and those with extreme risk attitudes. whyItMatters":"The findings highlight that LLMs can produce markedly different financial advice from a single, potentially unreliable risk‑tolerance response, underscoring the need for repeat‑testing and transparency in AI‑driven financial recommendations."
The paper examines how AI‑written research papers often contain misattributed numerical claims. By re‑analyzing 540 judgments from a benchmark of nine AI agents on fifteen research problems, the authors manually labeled 256 claims and found that most failures stem from misalignment between the paper and the underlying work rather than outright cheating. They propose a deterministic claim‑to‑evidence checker that achieves over 90% precision and, when combined with an LLM verifier, offers a more reliable audit of AI‑generated research.
The paper introduces Self-Improving Retrieval-Augmented Generation (RAG), a framework that splits document question answering into Retrieval, Reasoning, and Judge agents coordinated by an orchestrator. When the Judge scores an answer below a dynamic threshold, the system retries with broader retrieval, more careful prompting, and relaxed acceptance criteria, achieving 86% oracle-guided accuracy on FinanceBench with a 36.4% Lazarus Rate. The approach logs every decision with confidence scores, providing audit trails needed for regulated financial applications.
LiveSearch is a new financial benchmark that evaluates agents on up‑to‑date facts by storing each question as executable verifier code that calls a professional financial data service and extracts the answer at evaluation time. The gold answer is recomputed daily without human or language‑model intervention, ensuring that models must retrieve current data rather than rely on memorized, stale answers. The benchmark automatically generates these verifier programs from real tool responses, and experiments show that most questions change their gold value daily, revealing dynamic correctness that aggregate metrics can miss.
arXiv:2609.15319v1 Announce Type: cross Abstract: Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced a...
arXiv:2607. 17797v1 Announce Type: new Abstract: Financial statements (FS) such as Balance Sheet (BS), Income Statement (IS) and Cash-flow Statement (CS) summarize the annual financial performance of a company.