RATED (Reliability‑Aware Tool Retrieval via Execution Dynamics) is a retrieval framework that blends robust semantic matching with dynamic reliability adaptation for AI agents using external tools. It uses a frozen dual‑encoder with contrastive fine‑tuning for semantic relevance and a lightweight contextual bandit module that re‑ranks tool candidates based on real‑time execution feedback, allowing rapid adaptation to runtime failures without costly retraining. The EcoAgent benchmark, comprising 5,172 real‑world tools and over 210,000 query‑tool pairs, shows that RATED improves mean reciprocal rank by 52.2% over the ToolLLM API retriever and cuts fault selection from 86.2% to 2.9% during severe API outages, with negligible latency overhead.
arXiv:2609.35811v1 Announce Type: cross Abstract: Tool retrieval is a critical bottleneck for LLM-based agents operating over large, heterogeneous API ecosystems. Existing approaches face an inherent...
arXiv:2605. 02411v2 Announce Type: replace Abstract: A semantic gap separates how users describe tasks from how tools are documented.
The paper introduces FRESH, a Failure-aware Retrieval framework that uses Experience-Structured Heterogeneous graphs to transform past successes and failures into structured external memory for tool‑using agents. By explicitly modeling dependencies among tasks, actions, errors, repairs, and execution conditions, FRESH enables frozen language models to reuse reliable strategies, avoid recurring failures, and make safer decisions in stateful tool interactions. Experiments on τ‑Bench and AppWorld with multiple open‑source models demonstrate that FRESH consistently improves task success and tool‑use reliability compared to no‑memory agents and other memory‑based baselines.
AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.
arXiv:2609.06059v1 Announce Type: new Abstract: As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to a...
Bench2Robust is a framework that augments existing tool‑using LLM benchmarks with scenario‑controlled solvability, adding episodes that require retrying the original path, switching to a fallback tool, or recognizing that no viable path remains. It separates recovery‑policy quality from randomness in failure persistence, and across 7 models and 10 task slices, 69 of 70 model‑subset pairs degrade under tool failures by up to 46.7 percentage points. The authors introduce Bayesian Tool Memory (BTM) for structured runtime recovery context and use reinforcement learning (RL) for learned recovery behavior, achieving up to a 20.7‑point improvement over the base model without sacrificing clean performance.
arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.
arXiv:2607. 25718v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly rely on invoking external tools to complete real-world tasks.
arXiv:2610.23707v1 Announce Type: new Abstract: A multi-step agent task fails when a single tool it needs is missing, yet the tools an agent may need are too many to list: the roughly 8,200 tools on...
arXiv:2606. 12451v1 Announce Type: new Abstract: Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck.
arXiv:2608. 12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation.
arXiv:2610.08452v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is a widely used approach for grounding large language models (LLMs) in external knowledge. However, configuring...