arXiv:2609.35811v1 Announce Type: cross
Abstract: Tool retrieval is a critical bottleneck for LLM-based agents operating over large, heterogeneous API ecosystems. Existing approaches face an inherent...
By Zongze Wu, Yani Guo, Runnan Li
arXiv:2605. 02411v2 Announce Type: replace Abstract: A semantic gap separates how users describe tasks from how tools are documented.
By Kyle Zheng, Han Zhang, Renliang Sun, Chenchen Ye, Wei Wang
The paper introduces FRESH, a Failure-aware Retrieval framework that uses Experience-Structured Heterogeneous graphs to transform past successes and failures into structured external memory for tool‑using agents. By explicitly modeling dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH enables frozen language models to reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on τ‑Bench and AppWorld with multiple open‑source models demonstrate that FRESH consistently improves task success and tool‑use reliability compared to no‑memory agents and other memory‑based baselines.
By Jiaxing Li, Lei Song, Rui Dong, Youyong Kong
AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.
By Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth
arXiv:2609.06059v1 Announce Type: new
Abstract: As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to a...
By Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, Yanbing Liu, Pei Fu
Bench2Robust is a framework that augments existing tool‑using LLM benchmarks with scenario‑controlled solvability, adding episodes that require retrying the original path, switching to a fallback tool, or recognizing that no viable path remains. It separates recovery‑policy quality from randomness in failure persistence, and across 7 models and 10 task slices, 69 of 70 model‑subset pairs degrade under tool failures by up to 46.7 percentage points. The authors introduce Bayesian Tool Memory (BTM) for structured runtime recovery context and use reinforcement learning (RL) for learned recovery behavior, achieving up to a 20.7‑point improvement over the base model without sacrificing clean performance.
By Chaoran Chen, Vy Nguyen, Ziji Zhang, Abhinav Gullapalli, Ziyi Wang, Yuxuan Lu, Dakuo Wang, Jing Huang, Zhou Yu, Jin Lai