The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
The paper reports on applying AutoResearch—a large language model that iteratively edits training scripts—to optimize embedding systems for a book recommendation pipeline at production scale. Over twelve weeks, the authors ran 220+ experiments across two representation‑learning systems, uncovering five recurring failure modes (infrastructure fragility, agent memory decay, search‑direction stagnation, iteration‑cost asymmetry, and metric fixation) that were not present in smaller settings. They propose a three‑principle scaffolding (prevent, persist, redirect) to address these failures, achieving a 1.82× lift in Recall@6, a 2.1× lift in coherence, and an autonomous text‑only fallback that expanded catalog coverage by 5.8×.
By Aparajith Chandran, Juwon Kim, Saurav Jha, Pablo Castells, Florian Hottier
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
By Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran
The paper introduces the concept of an agent’s "taste"—its ability to make effective long‑horizon decisions—and presents Taste‑Bench, a new benchmark that automatically generates decision‑fork questions from agent trajectories. Taste‑Bench evaluates models on choosing the best path without seeing future outcomes, revealing that top models answer only about 60% of questions correctly and that later‑appearing evidence makes forks harder. The authors also demonstrate that training a student model to mimic a teacher’s judgment improves decision quality and overall success on held‑out software engineering tasks.
By Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.
By Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz, Michal Shmueli-Scheuer, Roi Reichart
AutoResearch is a two‑stage autonomous research system that links Idea Generation with Idea Execution. In the generation phase it blends new research signals with existing domain knowledge, identifies transferable mechanistic insights, and produces grounded, testable research plans through multi‑model generation and cross‑review. The execution phase then decomposes these plans into experiments, iteratively implements and diagnoses them, and uses independent evidence‑based review to accept or revise conclusions, thereby turning ideas into measurable progress while minimizing hallucinations.
By Yiming Ren, Xiang Liu, Qumeng Sun, Xiao Zhang, Jiahao Li, Haoyang Zhang, Junjie Wang
The paper re‑evaluates memory‑based self‑improving agents by adding multiple runs to measure variance and by randomizing task order. It finds that agent performance is noisy in complex, multi‑step environments and that improvement depends heavily on the sequence of tasks, revealing a hidden curriculum effect. The authors suggest that underspecification of tasks and environments contributes to this fragility and demonstrate that adding detailed rubrics and feedback can partially mitigate performance drops, though gaps remain.
By Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, Chien-Sheng Wu
The paper introduces ConflictGuide, a method that enhances LLM-based AutoResearch by incorporating feedback on competing behaviors during model code editing. By first exploring with scalar task performance and then using probes to measure and alleviate conflicts, ConflictGuide increases the proportion of edits that improve multiple behaviors and sustains progress beyond scalar-only plateaus. Experiments across five model families show reductions in task and conflict-related errors by up to 28% and 14% compared to scalar-only AutoResearch.
By Binqian Xu, Qiran Zou, Xiangbo Shu, Dianbo Liu
arXiv:2606. 28733v1 Announce Type: new Abstract: LLM agents are expected to act over multiple turns, using search, browsing interfaces, and terminal tools to complete user goals.
By Han Luo, Bingbing Wen, Lucy Lu Wang
arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.
By Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica
arXiv:2606. 01961v1 Announce Type: new Abstract: Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering.
By Junqi Liu, Salena Song, Yuhan Wang, Jiawei Mao, Hardy Chen, Xiaoke Huang, Tianhao Qi, Pengfei Guo, Yucheng Tang, Yufan He, Can Zhao, Andriy Myronenko, Dong Yang, Daguang Xu, Yuyin Zhou
The paper investigates whether a language‑model agent can autonomously study an unfamiliar environment without prior task instructions or examples, and decide how to prepare for future tasks. It formalizes task‑agnostic environment preprocessing, where a studying system explores under a budget to produce reusable artifacts for a later solver. Experiments on six diverse benchmarks show that a meta‑agent variant often outperforms fixed methods, though larger budgets do not consistently boost downstream reward, yet the artifacts still reduce test‑time sampling needed to achieve a target score.
By Vinay Samuel, Varun Ursekar, Vijay S. Kalmath, Apaar Shanker, Veronica Chatrath, Yuan Xue