BENCHCOMPASS is a new payment‑domain benchmark that transforms typed evidence packs into scenario‑grounded tasks, applies LLM‑based quality checks, generates attack variants, and reserves final item admission for domain experts. It includes an expert‑reviewed Pro benchmark covering payment knowledge, context‑grounded scenario reasoning, and attacked open robustness, plus a lower‑assurance Normal pool. Across 16 model variants, BENCHCOMPASS reveals distinct failure modes—missing payment knowledge, incomplete reasoning, and failure to reject invalid workflows—while the best model scores 89.6% on open context‑grounded reasoning and 81.7% under attacked inputs.
"whyItMatters":"The benchmark provides a structured way to isolate and evaluate specific weaknesses in LLMs for payment operations, a critical financial infrastructure where rules change rapidly and decisions depend on complex contextual factors."
By Sijie Dong, Wei Ren, Xuanwei Hu, Jiawei Luo, Zifan Wang, Xiaoyun Feng, Hui Cai, Lyuxin Xue, Peng Lu, Jianshe Li, Xin Zhang, Wei Wu
arXiv:2607. 23123v1 Announce Type: new Abstract: Existing evaluations of large language models cover knowledge, reasoning, coding, and tool use, but they rarely treat a verifiable deliverable produced within a constrained workflow as the unit of evaluation.
By Summer Sun (Shaqiu Community)
SWE‑Clar is a benchmark that tests coding agents’ ability to clarify incomplete client requirements before coding. It contains 100 real freelance software tasks from Upwork, each worth at least US$4,000, and includes multi‑round interactions, 512 clarification questions, and separate runs for code implementation and clarification assessment. Experiments with nine state‑of‑the‑art models show that even the best, GPT‑6 Astra, only passes 67% of tasks, highlighting the difficulty of autonomous software delivery in this context.
By Jialei Li, Dingxian Wang, Xiaoyan Zhao, Yimeng Bai, Xuanyi Zhu, Yang Zhang, Chao Yin, Wentao Shi, Ganesh Prabhu, Karthik Gomadam Rajagopal, Brett Levert, Darvin Yi, Andrew Rabinovich, Zhao Chen
The paper proposes a three‑engine architecture for reliable financial AI, combining a financial knowledge graph, a deterministic optimization engine, and a multi‑agent LLM layer. A coordination layer called context engineering selects and validates query‑specific context from ERP systems and other tools. Observations from 15 deployments show that multi‑agent validation reduces hallucination rates to below 1%, while reusable mappings and templates cut deployment time from six to two weeks.
By Pradeep Kumar GS, Ravi Tanniru
arXiv:2607. 06411v1 Announce Type: cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue.
By Evgeny Shilov (Independent Researcher)
arXiv:2608. 09072v1 Announce Type: cross Abstract: Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs.
By Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo
Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. Existing repository-level agentic benchmarks do not measure this setting: their task statements are English by design.
AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.
By Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu
arXiv:2609.06059v1 Announce Type: new
Abstract: As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to a...
By Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, Yanbing Liu, Pei Fu
RubricRefine is a training‑free pre‑execution refinement method that generates task‑specific rubrics from tool documentation, scores candidate code against explicit contract checks, and iteratively repairs failures before execution. It achieves an average score of 0.86 across seven models on M3ToolEval without any execution attempts, outperforming prior inference‑time baselines while incurring lower latency. The approach shows consistent performance on single‑step API‑Bank tasks and maintains an advantage in multi‑turn settings on AppWorld, with its effectiveness tied to the quality of the supplied documentation.
By Will LeVine, Brendan Evers, Sam Saltwick, Abhay Venkatesh
AgentScript‑Eval is a benchmark of 100 real tasks for Salesforce’s AgentScript DSL, testing LLMs on authoring, debugging, and optimizing conversational agents. Five models were evaluated zero‑shot, with the best achieving only 25.0% pass@3 and all scoring 0% on authoring from scratch. The study shows that model size and cost are poor predictors of success; instead, a compile‑and‑edit loop and the wording of domain guidance significantly improve performance, with the best result (37%) achieved by rewriting a skill within a live plugin.
By Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui
arXiv:2607. 25400v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly entrusted with natural-language workflow instructions (e.
By Jincheng Wang, Min Zheng, Tao Wei