TauIndianBankBench is a new benchmark that evaluates conversational agents on 1,000 Indian retail‑banking tasks by checking not only the final records but also whether every required procedural step was performed and no unrequested changes were made. It highlights that models can appear successful when judged only on final records, yet may skip necessary steps or alter records incorrectly. The benchmark provides a more accurate assessment of language model performance in the Indian banking domain.
By Krishanu Adhikary, Prashant Devadiga
arXiv:2607.20491v4 Announce Type: replace-cross
Abstract: When a financial agent repeats its decision, how much does that tell us about the actions behind it? DFAH-Bench examines this question throug...
By Raffi Khatchadourian
FINCOSMOS is an open‑world evaluation framework for financial language agents that tests their ability to acquire data from the Web, construct backtests within a data‑cutoff, and deliver verifiable results without any pre‑provided data or reference answers. In the benchmark, a simulated user hides a strategy and only reveals it upon request; agents must build their own backtests and pass 15 veto checks to be accepted. Across 150 sessions with four coding agents and a tool‑less model, no session was accepted, highlighting widespread failures and revealing that 88.33% of failures were attributed to the agents themselves.
By Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou
arXiv:2607. 25398v1 Announce Type: new Abstract: Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows.
By Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen
FinalityBench is an executable benchmark that tests how agents decide on shipping, re‑capturing, refunding, or waiting when a merchant’s payment processor, ledger, ERP, and bank feed receive delayed, duplicated, dropped, or reordered messages, causing contradictory beliefs about an order. The benchmark uses a hidden canonical event log and faulted delivery streams to generate system views, scoring each episode by the merchant’s terminal economic position relative to a privileged reference. It contains 321 tasks, including 45 twin pairs where all four views are identical yet the correct disposition differs, and evaluates nine programmatic policies, revealing that a ship‑on‑first‑sign policy performs best by accuracy but worst by paired loss, while a runtime‑gated irreversible‑action policy achieves 85.4% accuracy without losing money.
By Abhishek Sharma
arXiv:2608. 06144v1 Announce Type: new Abstract: Most agent benchmarks evaluate tasks independently and cannot measure whether experience from one task helps with later tasks.
By Bo Deng (Beihang University, Qwen DianJin Team, Alibaba Cloud Computing), Kang Zhou (Qwen DianJin Team, Alibaba Cloud Computing), Lifan Guo (Qwen DianJin Team, Alibaba Cloud Computing), Chongyang Tao (Beihang University), Xuanren Chen (Beihang University), Chenggang Xie (Beihang University), Renzhao Liang (Beihang University), Feng Chen (Qwen DianJin Team, Alibaba Cloud Computing), Chi Zhang (Qwen DianJin Team, Alibaba Cloud Computing)