The paper introduces Traverse, a benchmark of 2,518 agent trajectories and 6,967 annotated mistakes across software engineering, computer use, and science tasks, revealing that failures often go unrecovered and can cause irreversible harm before a run is deemed successful. It shows that human judges struggle to detect the first mistake in most runs, while a 4‑billion‑parameter verifier called Scout can locate failures more effectively and improve task success when used to select among candidate runs. The study demonstrates that making failure detection inexpensive and reliable can enable long‑horizon agents to learn from their own mistakes and increase trustworthiness in autonomous AI.
By Salman Rahman, Yubin Kim, Mihir Parmar, A. Ali Heydari, Genglin Liu, Simon A. Lee, Weizhi Zhang, Arian Hosseini, Ahmed A. Metwally, Yuzhe Yang, Baharan Mirzasoleiman, Xin Liu, Pavel Izmailov, Saadia Gabriel, Mark Malhotra, Shwetak Patel, Daniel McDuff, Hamid Palangi
arXiv:2610.02588v1 Announce Type: new
Abstract: Agents are increasingly given open-ended research tasks: discovering an empirical law from self-designed experiments, improving a heuristic whose optim...
By Chengyang Shi, Xianglin Ji, Jintao Huang, Jicheng Wang, Yifeng He, Jiachen Liu
The paper introduces Thinking‑in‑Bets, a training‑free technique where a language‑model agent records its key claims, supporting checks, and confidence before receiving user feedback. By consulting this private claim checklist when challenged, the agent preserves verified work and focuses on uncertain or newly contested parts, reducing sycophantic revisions. Experiments on five Gemini models across eight benchmarks show that pre‑reply claim records nearly eliminate false reversals in patch review, protect report drafts from mistaken pushback, and lower unnecessary edits in multi‑turn coding while improving accuracy and confidence in non‑interactive tasks.
By Nghi D. Q. Bui
LiveSearch is a new financial benchmark that evaluates agents on up‑to‑date facts by storing each question as executable verifier code that calls a professional financial data service and extracts the answer at evaluation time. The gold answer is recomputed daily without human or language‑model intervention, ensuring that models must retrieve current data rather than rely on memorized, stale answers. The benchmark automatically generates these verifier programs from real tool responses, and experiments show that most questions change their gold value daily, revealing dynamic correctness that aggregate metrics can miss.
By Sinuo Wang, Wentao Zhao, Jiacheng Lu, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
arXiv:2609.07603v1 Announce Type: new
Abstract: Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Age...
By Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie
AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.
By Shu Zhou, Ziyang He, Yuxin Rao, Yudong Fang, Jiali Qian, Dongyuan Chen, Juntao Zhang, Yong Hu