The paper introduces TURTLE, an Unsupervised Environment Design framework that evolves tool‑use tasks for AI assistants by using a proposer, solver, and feedback generator powered by large language models. TURTLE automatically generates self‑tuning, long‑horizon benchmarks that increase task difficulty, diversity, and failure modes across multiple applications. Experiments on AppWorld, τ‑Bench, and AutomationBench demonstrate significant gains in hard task percentages and broader reasoning challenges.
By Hyunji Alex Nam, Michael Dennis, Natasha Jaques, Minsuk Chang
Neuroscience data are fragmented across labs and formats, making reuse labor‑intensive. This study benchmarks coding agents on eight recent mouse neural recording papers, asking them to reformat data for a common decoder task. While agents handle many subtasks, they rarely produce fully error‑free solutions; the authors pair outcome metrics with a process rubric to evaluate errors and propose agent‑assisted data‑sharing practices.
By Ling-Qi Zhang, Kristin Branson
arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.
By Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz, Michal Shmueli-Scheuer, Roi Reichart
arXiv:2605.12894v2 Announce Type: replace-cross
Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...
By Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques
arXiv:2608. 14036v1 Announce Type: new Abstract: Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge.
By Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li
arXiv:2604. 00594v2 Announce Type: replace Abstract: As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult.
By Chris Ge, Daria Kryvosheieva, Daniel Fried, Uzay Girit, Kaivalya Hariharan
The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.
By Anjali Kantharuban, Jonas Mueller
EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.
By Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang
The paper introduces MAGE, a framework that uses user manuals to enhance computer use agents (CUAs) for new software. MAGE retrieves relevant manual content during execution, generates procedural guidance, and adapts the agent policy through interaction. In training, it converts manuals into executable task-plan pairs to guide rollouts, reducing the need for human‑annotated demonstrations and improving task success rates across six applications.
By Chulhoon Yoon, Kyungri Park, Jinyeong Yun, Daeyun Jeong, Sukmin Yun, Woohwan Jung
arXiv:2509.08494v2 Announce Type: replace-cross
Abstract: As humans delegate more tasks and decisions to artificial intelligence (AI), we risk losing control of our individual and collective futures....
By Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes, Jacy Reese Anthis
The paper introduces GenUI‑Harness, a multi‑agent system that pairs a Tool Agent for data retrieval with a GUI Coder Agent that generates structured interfaces to aid human‑agent interaction. It tackles training challenges by using Dynamic UX for lightweight reward collection and Reward Auditor to monitor reward quality. The authors present UI‑TAU Bench, a benchmark of 1,300 database‑backed tasks, and show that GenUI‑Harness improves Pass@3 scores and reduces dialogue rounds compared to existing models.
By Xiaolong Li, Xiaohan Xu, Jinyang Li, Xinnuo Xu, Ge Qu, Nan Huo, Jack Williams, Reynold Cheng
arXiv:2610.19560v1 Announce Type: cross
Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users....
By Rishitosh Kumar Singh, Amir Saeidi, Shamanthak Hegde, Shrinidhi Kumbhar, Chitta Baral