arXiv:2607. 22682v1 Announce Type: new Abstract: We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
The paper reports on applying AutoResearch—a large language model that iteratively edits training scripts—to optimize embedding systems for a book recommendation pipeline at production scale. Over twelve weeks, the authors ran 220+ experiments across two representation‑learning systems, uncovering five recurring failure modes (infrastructure fragility, agent memory decay, search‑direction stagnation, iteration‑cost asymmetry, and metric fixation) that were not present in smaller settings. They propose a three‑principle scaffolding (prevent, persist, redirect) to address these failures, achieving a 1.82× lift in Recall@6, a 2.1× lift in coherence, and an autonomous text‑only fallback that expanded catalog coverage by 5.8×.
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
The paper introduces the concept of an agent’s "taste"—its ability to make effective long‑horizon decisions—and presents Taste‑Bench, a new benchmark that automatically generates decision‑fork questions from agent trajectories. Taste‑Bench evaluates models on choosing the best path without seeing future outcomes, revealing that top models answer only about 60% of questions correctly and that later‑appearing evidence makes forks harder. The authors also demonstrate that training a student model to mimic a teacher’s judgment improves decision quality and overall success on held‑out software engineering tasks.
arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.
AutoResearch is a two‑stage autonomous research system that links Idea Generation with Idea Execution. In the generation phase it blends new research signals with existing domain knowledge, identifies transferable mechanistic insights, and produces grounded, testable research plans through multi‑model generation and cross‑review. The execution phase then decomposes these plans into experiments, iteratively implements and diagnoses them, and uses independent evidence‑based review to accept or revise conclusions, thereby turning ideas into measurable progress while minimizing hallucinations.