The study examines how different harness configurations affect a fixed large language model’s compliance with banking conduct rules. Four open‑weight models were tested across five harnesses on 156 pre‑registered retail‑banking scenarios, yielding 3,120 two‑turn episodes scored for rule violations and over‑refusals. Results show that while overall compliance rates varied modestly (87.3%–92.1%), the impact of a harness differed markedly between models, with some models’ best and worst harnesses differing by over 10 percentage points and a same‑model compliance reviewer reducing violations from 9.6% to 6.4%.
whyItMatters":"The findings suggest that validating a model’s compliance requires testing the specific model‑harness pair rather than the model alone, highlighting the importance of action logs over conversation transcripts for accurate conduct assessment."
By Mehmet Murat Huyut
arXiv:2607. 20491v1 Announce Type: new Abstract: Standard evaluation benchmarks measure what a tool-using agent decides, not whether it arrives at that decision through the same process each time.
By Raffi Khatchadourian
Bank-World is a benchmark comprising 50 banking tasks set in two simulated institutions, designed to evaluate agents on complex banking workflows that require more than correct calculations. The benchmark tests agents through single-turn assignments and multi-turn interactions with simulated employees, assessing their ability to reconcile records, obtain permissions, execute transactions, and maintain consistent instructions and status reports. Evaluation combines executable checks of recorded actions and state with model judgments of task requirements, revealing that agents often make correct decisions yet fail to complete work, with varying strengths across different models.
By Daniel Rupawalla
arXiv:2608. 09025v1 Announce Type: new Abstract: Financial agents can turn correct context into an unauthorized effect: a customer-facing commitment, trade, or deployed policy.
By Rui Tang, Qiangqiang Liu, Yichi Zhang, Youwei Wang, Xi Chen, Chen Dong
FinalityBench is an executable benchmark that tests how agents decide on shipping, re‑capturing, refunding, or waiting when a merchant’s payment processor, ledger, ERP, and bank feed receive delayed, duplicated, dropped, or reordered messages, causing contradictory beliefs about an order. The benchmark uses a hidden canonical event log and faulted delivery streams to generate system views, scoring each episode by the merchant’s terminal economic position relative to a privileged reference. It contains 321 tasks, including 45 twin pairs where all four views are identical yet the correct disposition differs, and evaluates nine programmatic policies, revealing that a ship‑on‑first‑sign policy performs best by accuracy but worst by paired loss, while a runtime‑gated irreversible‑action policy achieves 85.4% accuracy without losing money.
By Abhishek Sharma
arXiv:2608. 16055v1 Announce Type: new Abstract: Existing agent benchmarks ask whether the agent finished the task.
By Bowen Li, Guojun Wang
arXiv:2608. 13706v1 Announce Type: cross Abstract: Existing defenses against hallucination in retrieval-augmented and multi-agent pipelines remain partial: evidence is trusted despite modality disagreement, debate verifies an aggregate report rather than individual claims, and such verification occurs only after drafting, leaving inter-agent errors undetected until the final text.
By Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud, M. F. Mridha, Md. Alam Hossain
The paper audits self‑evolving financial agents—SkillOpt, Agent Workflow Memory (AWM), and ReasoningBank—by evaluating their performance, security drift, and execution‑interface mismatches in simulated e‑banking scenarios. It shows that while utility improves after evolution, exposure to malicious content and unauthorized state changes also rise, and that AWM’s text‑action envelope can disrupt tool execution, highlighting the need for comprehensive auditing beyond accuracy metrics.
By Jialong Li, Jialing Zhu
FinRiskAtlas is a Chinese-language benchmark designed to evaluate large language models (LLMs) for financial risk review by focusing on decision‑aligned tasks rather than generic financial knowledge. It contains 9,742 instances across 53 task families, including 42 domain‑knowledge families and 11 downstream review operations defined by explicit evaluation contracts. The extended FinRisk‑Ask framework replays 680 pre‑action states from 104 professional trajectories, withholding future evidence during inference to assess evidence‑state control and request targeting. Results across 33 model configurations show that operation‑level evaluation yields distinct rankings and that knowledge‑based shortlisting can incur significant regret, while frequent use of the Ask branch does not necessarily improve evidence acquisition, highlighting gaps in broad financial capability scores.
By Suyang Zhong, Jingzhe Zhu, Qi Xu, Liyao Sun, Yin Wang, Qingqing Sun, Shuai Chen, Tianyi Zhang
arXiv:2608. 19760v1 Announce Type: cross Abstract: Audited against causal ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals used to train LLM agents -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- identifies which steps causally matter better than chance.
By Haiyue Zhang
arXiv:2601.15322v3 Announce Type: replace-cross
Abstract: Tool-using agents can repeat a final decision while changing their recorded execution. We introduce the Determinism-Faithfulness Assurance Ha...
By Raffi Khatchadourian
arXiv:2610. 32380v1 Announce Type: new Abstract: Large language models (LLMs) can read startup pitch decks, but letting them emit investment scores directly invites score inflation, run-to-run variance and rationales that are hard to audit.
By Ming Ze Tang, David Cutting, Boyd Carson, Vasiliki Carson