The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
The paper investigates how coordination among AI agents serving different users degrades performance compared to a single coordinating agent. Across five advanced models and 77 scenarios in four shared-resource environments—API key budgets, clinic calendars, personal assistant bookings, and merge queues—the study finds that multi‑agent teams consistently underperform, sometimes collapsing entirely, and that even with communication channels coordination overhead remains significant. The authors identify specific failure modes such as stalling, action overriding, and claim fabrication, and propose environment‑specific mitigations like team leads and procedural instructions, while releasing the MAMUBench benchmark for future research.
By Sahan Paliskara, Nattaput Namchittai, Andrew Lampinen
arXiv:2609.40284v1 Announce Type: cross
Abstract: Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on...
By Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh
The paper introduces $ au^ au$-Bench, a benchmark that turns the construction of AI agents into a measurable task. In this environment a developer agent receives real business records, client requirements, a production API, an existing codebase, and constraints on cost and models, and must deliver a complete customer‑service agent. The benchmark evaluates performance by deploying the agent against simulated users, revealing that current state‑of‑the‑art models achieve only 23.9% success while an expert‑written reference scores 82.2%.
By Quan Shi, Keshav Dhandhania, Karthik Narasimhan, Victor Barres
arXiv:2607.10526v5 Announce Type: replace
Abstract: Self-improving personal agents write profiles, memories, and reusable skills that carry over from one chat to the next. Prior works ask whether a m...
By Xutao Mao, Liangjie Zhao, Leyao Wang, Rui Qian, Qiang Huang, Wentao Wang, Bo Han, Xiang Zheng, Cong Wang
The study evaluates whether multi‑agent systems (MAS) outperform single‑agent systems (SAS) when agents interact with users under a written policy on the dual‑control benchmark. Across 7,950 simulated conversations in airline, retail, and telecom domains, the MAS performed worse in 12 of 13 valid cells and tied only once, with a negative average gain of 28 points. Even in scenarios where prior theory predicted a benefit, the MAS lost performance and incurred higher token costs.
By Ethan Yang, Xiangcheng Shi