The paper introduces Experience Memory Graph (EMG), a framework that transforms agent failure recovery into a graph matching problem. By converting failed and successful trajectories into directed action decision graphs, EMG extracts common subgraphs and edit paths that reveal explicit correction strategies, storing them in a memory graph with intra‑task nodes and cross‑task edges. During testing, EMG retrieves relevant insights to guide the agent in a single attempt, improving performance on ALFWorld and ScienceWorld without extra reflection rounds.
By Wenjun Wang, Yuchen Fang, Fengrui Liu, Zibo Liang, Kai Zheng
EvoIn is a fine‑tuning framework that improves agent decision‑making by evolving new procedures from execution traces, validating them in the harness, and then rewriting the traces to internalize the logic into the model. The approach removes reliance on the evolved harness at inference time, enabling stronger decision‑making that generalizes to unseen tasks. Experiments show consistent gains, raising in‑domain pass rates by 10.9 points and out‑of‑domain by 9.2 points, and the method works across different model families.
By Shihan Dou, Shaofan Liu, Zhonghang Lu, Jiahang Lin, Shichun Liu, Binghai Wang, Jiajie Jin, Guanting Dong, Tao Gui, Qi Zhang, Xuanjing Huang
Concordat is a Python library that enforces typed session protocols on large language model agents, ensuring that agents only use tools and make decisions allowed by a finite‑state protocol. By restricting tool calls, requiring explicit human confirmation, and managing budgets, Concordat prevents agents from bypassing safety rules at runtime. In tests on tau‑bench airline tasks, Concordat agents never violated confirmation rules while maintaining comparable task success to standard agents.
By Milad Sikaroudi
The paper introduces PIP-ToM, a framework that addresses the evidence-admissibility problem in multimodal Theory of Mind (ToM) by ensuring that only evidence accessible to an agent and diagnostic of its mental state is used in hypothesis evaluation. PIP-ToM builds agent‑specific evidence stores, aligns them with structured mental‑state claims, and calibrates Bayesian inference scores while preserving the underlying inference process. Using GPT‑4o, the method outperforms existing ToM approaches on MuMA‑ToM, MMToM‑QA, and Hi‑ToM benchmarks, achieving accuracies of 92.56%, 95.00%, and 95.00% respectively.
By Ziqi Liao, Jiaqi Shen, Yiming Na, Kedi Lyu, Shao-Yuan Lo, Hongxia Xie
LiveSearch is a new financial benchmark that evaluates agents on up‑to‑date facts by storing each question as executable verifier code that calls a professional financial data service and extracts the answer at evaluation time. The gold answer is recomputed daily without human or language‑model intervention, ensuring that models must retrieve current data rather than rely on memorized, stale answers. The benchmark automatically generates these verifier programs from real tool responses, and experiments show that most questions change their gold value daily, revealing dynamic correctness that aggregate metrics can miss.
By Sinuo Wang, Wentao Zhao, Jiacheng Lu, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
CohortFlow is a steerable agent for clinical cohort curation that applies natural language corrections as localized updates to a structured specification, allowing selective invalidation of affected computations while keeping valid results. In experiments on 100 clinical‑trial eligibility scenarios, it achieved a micro‑F1 of 77.0% versus 76.4% for full regeneration, a non‑significant difference, but reduced correction latency by 64.8% after completion and 44.3% during execution, and cut model calls by 38.1%. A proof‑of‑concept run on MIMIC showed all 13 cohort states executed successfully, with corrections preserving unaffected computations and recomputing only changed or dependent nodes.
By Neil Chen, Adrian Serapio, Hanxue Gu, Evan Lee, Kang Wang, Yang Yang
Skill Dreaming is a framework that learns from pull request (PR) streams to build and continuously refine a repository‑specific skill library. By extracting executable knowledge from validated PRs, it guides LLM agents in implementing features, maintaining tests, documentation, and repository conventions. The framework integrates PR analysis, skill management, code execution, and continuous integration to create an end‑to‑end workflow that updates skills as new PRs arrive.
By Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner, Bowen Zhu, Liang Zhao
The paper introduces kw-bench, a benchmark that evaluates coding-agent harnesses on investigative knowledge work, specifically a five‑year VAT reconstruction task across multiple SAP systems. It contrasts generative benchmarks with this real‑world, evidence‑heavy task, showing that model‑agnostic harnesses performed well while lab harnesses failed. The study demonstrates how frontier models can dramatically reduce time and cost for complex audit‑style engagements.
By George Juraj Salapa
RouteDRAG is a learning-based framework for multimodal document retrieval-augmented generation that learns when to invoke an agent and how to control its execution. It comprises a unified document index, adaptive path routing, dynamic agent execution, and sequential policy learning to balance accuracy and computational cost. Experiments on two benchmarks show a 3.47‑point accuracy gain over shorter‑path-only methods while using the agent for only 79.3% of queries, outperforming uniform agent execution with fewer answering calls.
By Zekai Chen, Hao Zhu, Yu Liu, Xunkai Li, Zejian Kang, Rui Qian, Quanjiang Li, Rong-Hua Li
The paper studies when to intervene in large language model (LLM) agents that interact with web services. Using the ReInBench dataset of 1,259 agent traces and 1,403 error events, the authors find that 725 errors self‑recover, often within a few steps, and that self‑recovery can aid subsequent error handling and overall task completion. They propose ReCoF, a predictor that decides whether an error will self‑recover, achieving a 9.02% relative improvement in Macro‑F1 over the best baseline.
By Chiming Duan, Tong Jia, Yingxi Chen, Minghua He, Lingzhe Zhang, Ying Li
The paper argues that classification accuracy alone is insufficient for evaluating AI agent restart monitors. It introduces a decision‑value framework with a decision‑sufficiency criterion and a finite‑sample decision audit to assess whether restarting an agent is beneficial. Empirical results on SymPy issues show that high accuracy can lead to both gains and losses, and that many traces fail to meet the audit’s certification threshold.
By Siddharth Jain, Venkat N. Vedam
The paper introduces Population-Centered Evolution (PCE), a new principle for improving self‑improving AI agents by leveraging patterns of success, failure, and prior modifications across the entire population rather than focusing solely on selected parent agents. PCE distills task records and modification histories into a population block that guides enhancements to a chosen parent’s harness. Experiments show that PCE raises success rates on SWE‑bench Verified and Polyglot benchmarks, narrowing the gap between population coverage and best‑agent performance.
By Shuo Yan, Ningning Hou, Haoyu He, Haochen Pei, Shang Gao
The paper introduces ChronoFake, a benchmark of 30,706 news articles that isolates temporally disordered fake news, where factual events are rearranged into false chronologies. It identifies the "evidence coherence trap," where verification relies on locally coherent but temporally mismatched evidence. To counter this, the authors propose DelSA, a document-selection agent that iteratively refines evidence during training and uses a frozen rulebook for inference, leveraging an LLM for evidence characterization and a verifier-trained reranker for timeline consistency, achieving superior performance on ChronoFake and competitive results on other datasets.
By Chi Wang, Min Gao, Yifei Zhang, Bincheng Gu, Zongwei Wang, Wentao Li
The paper extends the GEPA optimizer to generate and evaluate multiple candidates per optimization step, rather than a single candidate. Experiments on two tasks show that batched runs cut wall‑clock time by roughly half or more, while also improving held‑out test scores (e.g., LiveBench‑Math from 68.9% to 72.1% and HoVer from 49.0% to 60.0%). A simple runtime model explains the speedups under a finite worker pool, and batched settings use metric‑call and dollar budgets more efficiently.
By Jialin Zhang, Lakshya A Agrawal, Shangyin Tan, Donghyun Lee, Lutfi Eren Erdogan, Dan Klein, Koushik Sen, Alex Dimakis, Matei Zaharia
The paper introduces RuHarness, a framework that co‑evolves rubrics and harnesses to enable continual self‑improvement of autonomous agents that rely on large language models. RuHarness automatically discovers and reassesses behavioral criteria from execution trajectories, prioritizes unresolved weaknesses, and uses these insights to guide targeted harness updates while keeping the task objective unchanged. Experiments across varied agent tasks and backbone models demonstrate that RuHarness outperforms existing harness‑evolution baselines, especially on unseen held‑out tasks, and that its adaptive rubric prioritization supports sustained improvement and stable generalization over successive evolution rounds.
By Hao Chen, Yukun Yan, Ziyu Han, Ye He, Chuanzhe Guo, Shangzhan Li, Qi Shi, Qingfu Zhu, Maosong Sun, Wanxiang Che
Evolving Agent Teams (EAT) is a multi‑agent framework designed to overcome local optima that plague existing coding agents when optimizing GPU kernels. It employs a central supervisor to diagnose and restructure agent teams and a persistent file system that records learning documents, evolution history, and team artifacts. Across a range of benchmarks—including FlashInfer, FlashKDA, and RoboDojo—EAT consistently outperforms strong coding‑agent baselines, achieving up to 1.2× speedup on FlashInfer kernels and superior results on several other tasks.
By Shiyi Cao, Ziming Mao, Dacheng Li, Joseph E. Gonzalez, Ion Stoica
RubricAtlas is a framework that extracts type-level analytical rubrics from expert reports to evaluate deep research agents. It organizes binary checkpoints around recurring analytical questions, linking them to report passages and assessing completeness separately from argument validity. Experiments on 3,079 Chinese financial reports show that individual reports cover only about half of their type’s questions, and that using different reports for requirements changes evaluation scores and agent rankings.
By Delin Wang, Sinuo Wang, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai
Proactive Memory for Event-Driven LLM Agents introduces ProactMem, a system that organizes an agent’s history into recent events, temporal trajectories, and behavioral regularities. It validates trajectory facts against source events and assesses regularities through historical replay before matching them to current conditions. The authors also present ProactiveStream, a longitudinal benchmark with three application scopes, and report that ProactMem achieves an overall F1 of 52.6%, a 21.4% relative improvement over the strongest of five external baselines.
By Jinyang Li, Zichuan Fu, Xian Wu, Wenlin Zhang, Yimin Deng, Kaifeng Guo, Guojing Li, Junjia QI, Xiangyu Zhao
AutoVend is a benchmark that tests large language model agents by having them manage 20 vending machines across five site types over 180 simulated days via a web interface. The simulator incorporates realistic logistics such as shared warehouse inventory, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. Five LLM agents were evaluated against a rule‑based baseline, revealing that most agents underperformed the baseline and that higher inference costs can distort performance rankings.
By Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai
The paper introduces BiPA, a method for evaluating skill optimizers that rewrite natural‑language procedures for agents. By replaying the optimizer’s acceptance rule on identical candidates and recording episodes as bipartite graphs of turns and read text, BiPA isolates the effect of selection from the cost of added input tokens. Experiments on three optimizers, three benchmarks, and two models show that learned skills perform only marginally better than hand‑written seeds, and that apparent dev‑set gains often disappear on test, revealing that optimizers tend to add redundant rules that increase token usage.
By Jiawei Zhou, Bo Peng, Yuxuan Huang, Ming Zhang, Youbang Sun, Xiang Wang, Chaochao Lu