arXiv Machine Learning
DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
1d ago
The Harness Is the Model: Agent Scaffolding Changes a Fixed LLM's Compliance with Banking Conduct Rules, Unevenly Across Models
The study examines how different harness configurations affect a fixed large language model’s compliance with banking conduct rules. Four open‑weight models were tested across five harnesses on 156 pre‑registered retail‑banking scenarios, yielding 3,120 two‑turn episodes scored for rule violations and over‑refusals. Results show that while overall compliance rates varied modestly (87.3%–92.1%), the impact of a harness differed markedly between models, with some models’ best and worst harnesses differing by over 10 percentage points and a same‑model compliance reviewer reducing violations from 9.6% to 6.4%. whyItMatters":"The findings suggest that validating a model’s compliance requires testing the specific model‑harness pair rather than the model alone, highlighting the importance of action logs over conversation transcripts for accurate conduct assessment."
arXiv AI
Jul 24
DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making
arXiv:2607. 20491v1 Announce Type: new Abstract: Standard evaluation benchmarks measure what a tool-using agent decides, not whether it arrives at that decision through the same process each time.
arXiv AI
6d ago
Bank-World: A Benchmark for Evaluating Agents on Banking Workflows
Bank-World is a benchmark comprising 50 banking tasks set in two simulated institutions, designed to evaluate agents on complex banking workflows that require more than correct calculations. The benchmark tests agents through single-turn assignments and multi-turn interactions with simulated employees, assessing their ability to reconcile records, obtain permissions, execute transactions, and maintain consistent instructions and status reports. Evaluation combines executable checks of recorded actions and state with model judgments of task requirements, revealing that agents often make correct decisions yet fail to complete work, with varying strengths across different models.
arXiv AI
Aug 11
Context Is Not Authority: Structured Runtime Governance for Financial Market Agents
arXiv:2608. 09025v1 Announce Type: new Abstract: Financial agents can turn correct context into an unauthorized effect: a customer-facing commitment, trade, or deployed policy.
arXiv AI
Sep 7
FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality
FinalityBench is an executable benchmark that tests how agents decide on shipping, re‑capturing, refunding, or waiting when a merchant’s payment processor, ledger, ERP, and bank feed receive delayed, duplicated, dropped, or reordered messages, causing contradictory beliefs about an order. The benchmark uses a hidden canonical event log and faulted delivery streams to generate system views, scoring each episode by the merchant’s terminal economic position relative to a privileged reference. It contains 321 tasks, including 45 twin pairs where all four views are identical yet the correct disposition differs, and evaluates nine programmatic policies, revealing that a ship‑on‑first‑sign policy performs best by accuracy but worst by paired loss, while a runtime‑gated irreversible‑action policy achieves 85.4% accuracy without losing money.
arXiv AI
Aug 18
Governance at the Boundary: How Agent Decomposition Degrades Policy Compliance
arXiv:2608. 16055v1 Announce Type: new Abstract: Existing agent benchmarks ask whether the agent finished the task.