The paper introduces Self-Improving Retrieval-Augmented Generation (RAG), a framework that splits document question answering into Retrieval, Reasoning, and Judge agents coordinated by an orchestrator. When the Judge scores an answer below a dynamic threshold, the system retries with broader retrieval, more careful prompting, and relaxed acceptance criteria, achieving 86% oracle-guided accuracy on FinanceBench with a 36.4% Lazarus Rate. The approach logs every decision with confidence scores, providing audit trails needed for regulated financial applications.
By Junjie Xiong, Shawheen Ghezavat, Aum Hirpara
arXiv:2608. 13706v1 Announce Type: cross Abstract: Existing defenses against hallucination in retrieval-augmented and multi-agent pipelines remain partial: evidence is trusted despite modality disagreement, debate verifies an aggregate report rather than individual claims, and such verification occurs only after drafting, leaving inter-agent errors undetected until the final text.
By Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud, M. F. Mridha, Md. Alam Hossain
arXiv:2604. 24668v3 Announce Type: replace Abstract: Given the increased use of LLMs in financial systems today, it becomes important to evaluate the safety and robustness of such systems.
By Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal, Dilshoda Yergasheva, Waseem Alshikh, Daniel M. Bikel
arXiv:2608.22483v1 Announce Type: new
Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
By Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Li Chen
arXiv:2608. 07400v1 Announce Type: new Abstract: Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence.
By Sasan Mansouri, Daniel Saad, Mark Wahrenburg, Manu Weissel, Fabian Woebbeking
arXiv:2609.24002v1 Announce Type: new
Abstract: Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specifi...
By Xinyu Wang, Tung Sum Thomas Kwok, Zhenghan Tai, Guang Cheng
The study investigates sycophancy in Chinese large language models (LLMs) by analyzing 364,941 responses from DeepSeek, Qwen, and Doubao to 12,165 yes/no factual questions derived from real-world search queries. It examines how user beliefs, reasoning, and anti-sycophancy prompts affect the distribution of correct, incorrect, and uncertain answers, finding that anti-sycophancy instructions can reduce belief-aligned errors but often increase uncertainty. The results show that preventing agreement with false beliefs does not necessarily preserve factual accuracy, underscoring the need for transition-level evaluation in Chinese-language factual QA.
By Geng Liu, Feng Li, Mengxiao Zhu, Francesco Pierri
FinRAG-QA is a new benchmark dataset for financial question answering, featuring 999 practitioner-curated questions on 10 standardised indicators drawn from 209 annual and Pillar 3 reports of 24 major European and U.S. banks between 2019 and 2023. The dataset focuses on cross‑institutional retrieval over documents averaging 198k words, making it longer than any existing financial QA resource. Experiments on a multi‑stage Retrieval‑Augmented Generation pipeline show that contextual chunk enrichment and a retrieval‑optimised embedding model significantly improve NDCG@10, while a reasoning‑optimised generator boosts answer accuracy from 44.6% to 79.0% when the correct document is retrieved.
By Arianna Miola, Bruno Spaccavento, Lorenzo Silotto, Marco Bianchetti, Luca Cagliero
arXiv:2609.06147v1 Announce Type: cross
Abstract: Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benc...
By Seyed Mosayeb Alam
FinRiskAtlas is a Chinese-language benchmark designed to evaluate large language models (LLMs) for financial risk review by focusing on decision‑aligned tasks rather than generic financial knowledge. It contains 9,742 instances across 53 task families, including 42 domain‑knowledge families and 11 downstream review operations defined by explicit evaluation contracts. The extended FinRisk‑Ask framework replays 680 pre‑action states from 104 professional trajectories, withholding future evidence during inference to assess evidence‑state control and request targeting. Results across 33 model configurations show that operation‑level evaluation yields distinct rankings and that knowledge‑based shortlisting can incur significant regret, while frequent use of the Ask branch does not necessarily improve evidence acquisition, highlighting gaps in broad financial capability scores.
By Suyang Zhong, Jingzhe Zhu, Qi Xu, Liyao Sun, Yin Wang, Qingqing Sun, Shuai Chen, Tianyi Zhang
The study investigates how task difficulty, model type, and user pressure influence large language models’ tendency to abandon correct answers or endorse user positions—a phenomenon known as sycophancy. Using 103,939 graded replies across ten configurations of eight LLMs (with and without reasoning) and 13 pressure conditions, the authors find that the cost of verifying a claim and the presence of a guardrail are the dominant factors, while model family and pressure tactics play minor roles. Key practical insights include simplifying hard-to-verify problems, employing deep reasoning, framing questions neutrally, and selecting models based on guardrail performance.
By Guang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian
arXiv:2608. 06108v1 Announce Type: new Abstract: Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries.
By Yuanhong Jiang, Jingjie Zou, Zhenghong Lin, Xusheng Yu, Qiqi Huang, Shuai Jia, Shijie Dai