arXiv AI

Trustworthy FinAInce: Unpacking How AI-Mediated Financial Advice is Judged

The study investigates how people evaluate AI-generated financial advice by conducting a randomized vignette experiment with 285 U.S. adults. Participants were presented with consistent financial recommendations delivered in three styles—AI, expert, and online community—alongside source labels. The results show that advice style most strongly influenced message and safety appraisals, expert labels increased perceived source knowledge, and decision context shaped risk and safety judgments, with these appraisals explaining a large portion of overall quality, trust, and intended reliance.

arXiv AI
Oct 15

FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming

FinRED is an expert‑guided framework for generating and evaluating financial LLM red‑teaming tests. It maps global financial standards to specific threats, converts real documents into context‑rich prompts, and offers a validated rubric that reduces false negatives. The framework is deployed in South Korea’s Financial Security Institute sandbox and is gated for qualified researchers.

By Daeyoung Park, Chaeyun Kim, Junghwan Kim, Jinyoung Jeong, Eunji Song, Yongtaek Lim, Minwoo Kim
arXiv AI
Aug 19

When Personalization Becomes Bias: Structural and Discursive Religious Framing in AI-Generated Financial Advice

The study examines how large language models (ChatGPT, Gemini, and Grok) embed religious bias in AI‑generated financial advice. Using 432 simulated advisor‑client interactions across four religious identities and three financial decisions, the authors find that only 12‑18% of advice is unbiased, with Gemini showing the most bias and ChatGPT comparable to Grok. The research identifies structural biases in model design and discursive mechanisms—such as religious anchoring and tone modulation—that vary by scenario, revealing a tension between personalization and neutrality in AI advisory services.

By Muhammad Salar Khan, Hamza Umer, Hasan Mahmud, Sandra Rothenberg
arXiv Machine Learning
Aug 4

From Information to Delegation: Mapping Human-AI Financial Decision Making

arXiv:2608. 02100v1 Announce Type: cross Abstract: As AI increasingly participates in human decision making, understanding how decision-making authority is distributed between humans and AI has become a fundamental behavioural question.

By Iman Munire Bilal, Yingcan Carol Wang, Ajan Raj, Filippo Giovagnini, Pranav Tewari, Yuwei Zhang, Mei-Chen Zoe Liou, Qamar Zaman
arXiv AI
Nov 5

AgenticFinBench: Benchmarking Financial Agents on Diverse Tasks from Real-World Requests

AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.

By Shu Zhou, Ziyang He, Yuxin Rao, Yudong Fang, Jiali Qian, Dongyuan Chen, Juntao Zhang, Yong Hu
arXiv AI
Nov 5

Pairing Expert Critique with LLM-Driven Refinement: An Iterative Approach for Curating Quality AI Evaluation Datasets

The paper introduces SME-C, a human‑AI workflow that combines lightweight expert critiques with LLM‑driven refinement to create high‑quality, domain‑specific evaluation datasets and rubrics. In a financial business brief domain, SME‑C gathered 3,853 critiques from 411 professionals and used two LLM refinement strategies—one guided by written critiques and another by preference votes—to improve response quality. Blind validation with 100 additional professionals showed both strategies significantly outperformed the experts’ top‑ranked baseline, with critique‑based refinement yielding more domain‑grounded improvements and vote‑based refinement better at removing irrelevant content.

By Honson Ling, Kidus Admassu, Stephanie Rosenthal
arXiv AI
Oct 27

FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents

FinAutoRubric is a system that lets experts define reusable evaluation guidance for financial research agents, which the agents then use to automatically generate, review, and validate task‑specific rubrics. The approach replaces costly fixed rubrics with query‑specific ones that incorporate expert standards and can be updated as new information becomes available. On three expert‑authored finance benchmarks, the automatically generated rubrics closely match human scoring and are preferred by in‑house analysts in blind reviews.

By Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim, Daekyung Park, Sumin Kim, Jihoon Kwon, Jasmine Jia Geng, Andrew Chin, Yin Luo, Edward Tong, Yu Yu, Zach Golkhou, Minkyu Kim, Igor Halperin, Young Cha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee
arXiv AI
Oct 2

A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models

The paper introduces a benchmark for evaluating large language models on trustworthy analysis of earnings call transcripts. It proposes a numeric evidence evaluation method that assesses groundedness without expert annotation, and presents an automated pipeline that builds the ECTs-100 dataset from the top 100 S&P 500 constituents. The study also explores the failure mode of conscious incompetence, where models must recognize insufficient evidence and avoid hallucinations, finding that while groundedness is strong, correctness remains a challenge.

By Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo, Li Zhang, Zheng Ma
Hugging Face Trending Papers
Oct 29

Beyond LLM Wrappers: Context Engineering and the Three-Engine Architecture for Reliable Financial AI

The paper proposes a three‑engine architecture for reliable financial AI, combining a financial knowledge graph, a deterministic optimization engine, and a multi‑agent LLM layer. A coordination layer called context engineering selects and validates context from ERP systems, tools, and tenants to ensure numerical accuracy, traceability, and auditability. Observations from 15 deployments show that graph‑backed multi‑agent validation can reduce hallucination rates from 7‑9% to below 1%, while reusable mappings and validation policies cut deployment time from six to two weeks.