arXiv AI
Jul 29
GAUGE: Grading Agent-Built Financial Models Without a Golden Answer
arXiv:2607. 24889v1 Announce Type: cross Abstract: Financial models combine public disclosures with analyst assumptions to produce forecasts and valuations.
FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
FinAutoRubric is a system that lets experts define reusable evaluation guidance for financial research agents, which the agents then use to automatically generate, review, and validate task‑specific rubrics. The approach replaces costly fixed rubrics with query‑specific ones that incorporate expert standards and can be updated as new information becomes available. On three expert‑authored finance benchmarks, the automatically generated rubrics closely match human scoring and are preferred by in‑house analysts in blind reviews.
Related stories
arXiv AI
Aug 6
FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables
arXiv:2608. 04077v1 Announce Type: new Abstract: Evaluating financial AI agents requires criteria aligned with real professional work.
arXiv AI
Oct 12
Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics
The paper introduces Hindsight Structured Advice Distillation (HSAD), a method that trains a small advisor model to generate structured guidance for a frozen executor language‑model agent. By analyzing executor failures on a training set, a teacher writes advice that would have helped, and the advisor learns to produce this advice from input alone, adding minimal latency. Experiments on the FrontierFinance benchmark show that an 8B advisor boosts the qualification rate of Samaya’s research agent from 52.9% to 56.6%, surpassing the agent’s high‑effort configuration while cutting cost per query by 37%, and the approach also improves performance on WebShop, BIRD text‑to‑SQL, and ExpertLongBench.
arXiv AI
1d ago
RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents
arXiv:2610.36878v1 Announce Type: new Abstract: Deep research agents conduct multi-step Web searches and synthesise retrieved information into analytical reports. Evaluating these outputs requires ch...
arXiv AI
Aug 7
FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows
arXiv:2608. 06144v1 Announce Type: new Abstract: Most agent benchmarks evaluate tasks independently and cannot measure whether experience from one task helps with later tasks.
arXiv AI
Aug 13
FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents
arXiv:2608. 11683v1 Announce Type: new Abstract: AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow.
arXiv AI
Aug 20
FinSkillBench: Evaluating AI Agents and Domain Skills for Investment Management
FinSkillBench is an evaluation suite that tests whether language model agents can use financial domain skills to solve investment management tasks across portfolio construction, risk management, and fundamental analysis. The benchmark contains 12 subtasks with 2,603 episodes, each providing point‑in‑time inputs, hidden ground truth, and a verifier. Experiments show that curated skill packages improve performance significantly, while self‑generated skills offer little benefit, indicating that reliable procedural skills are crucial for effective AI agents in this domain.
Hugging Face Trending Papers
Aug 12
FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents
AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand.
arXiv AI
Oct 5
Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows
The paper introduces FinSkillBench, a benchmark of 2,603 financial AI agent episodes across portfolio construction, risk management, and fundamental analysis, evaluated with deterministic verifiers. Experiments on 9 models and 3 resource settings show that curated skill packages improve performance by +16.2 points, while in‑episode skill generation adds only +0.5 points. Decomposing the premium reveals that executable tools contribute +19.5 points, documentation +5.6 points, and their combination is subadditive, with the effect varying by workflow type.
arXiv AI
Jun 3
Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning
arXiv:2606. 03918v1 Announce Type: new Abstract: AI agents can increasingly handle the mechanical tasks of financial analysis: retrieving documents, calculating formulas, updating spreadsheets.
arXiv AI
Jun 19
FFinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming
arXiv:2606. 19887v1 Announce Type: cross Abstract: Existing safety benchmarks target general adversarial scenarios but miss finance-specific risks.
arXiv Computation and Language
Aug 27
FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review
FinRiskAtlas is a Chinese-language benchmark designed to evaluate large language models (LLMs) for financial risk review by focusing on decision‑aligned tasks rather than generic financial knowledge. It contains 9,742 instances across 53 task families, including 42 domain‑knowledge families and 11 downstream review operations defined by explicit evaluation contracts. The extended FinRisk‑Ask framework replays 680 pre‑action states from 104 professional trajectories, withholding future evidence during inference to assess evidence‑state control and request targeting. Results across 33 model configurations show that operation‑level evaluation yields distinct rankings and that knowledge‑based shortlisting can incur significant regret, while frequent use of the Ask branch does not necessarily improve evidence acquisition, highlighting gaps in broad financial capability scores.