The paper audits silent failures in agent-to-tool interactions within the ToolUniverse environment, focusing on 15 scientific tools. It identifies 91 failures—primarily missing data or inconsistent search/filtering—occurring mainly in the API and wrapper layers, and shows how these silent failures can propagate downstream into seemingly valid outputs. The authors propose contextual reliability and recommend testing, disclosure, monitoring, and measurement strategies for the agent-tool pipeline.
arXiv:2610.27777v1 Announce Type: new Abstract: Tool-using LLM agents increasingly answer e-commerce queries through multi-step tool interactions. The expected output of a tool call can shift between...
arXiv:2610.17418v1 Announce Type: cross Abstract: Agentic workflows combine large language models with planning, memory, tools, execution environments, and iterative feedback. Their failures are diff...
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
arXiv:2610.16333v1 Announce Type: new Abstract: As AI agents increasingly act through external tools, tool results are no longer simple outputs but execution artifacts that must remain correctly asso...
ToolRobustBench is a stage-wise diagnostic benchmark designed to evaluate and diagnose failures in tool‑calling agents, which are large language models that select tools, provide structured arguments, and interpret tool feedback. The benchmark aligns four perturbation families—tool‑interface, user‑intent, tool‑output/observation, and runtime‑environment—with the tool‑use pipeline, attributing failures to specific stages such as tool selection, schema grounding, argument binding, and feedback handling. Experiments across 15,456 instances, 7 models, and 16 local tools reveal that while overall performance is high, robustness degrades significantly, especially under tool‑output/observation perturbations, and mixed‑family perturbations produce non‑additive failure patterns.
arXiv:2606. 30531v1 Announce Type: new Abstract: Tool-augmented language-model agents are often evaluated by whether they select the correct tool, produce valid API arguments, and complete the requested task.
The paper introduces BenchGuard, a framework that uses advanced language models to audit execution-based agent benchmarks. BenchGuard systematically cross-verifies benchmark artifacts through structured LLM protocols, optionally incorporating agent solutions or execution traces for deeper diagnostics. In tests on ScienceAgentBench and BIXBench Verified-50, BenchGuard uncovered numerous author-confirmed and expert-identified issues, including fatal errors that made tasks unsolvable, and demonstrated cost-effective auditing for complex bioinformatics tasks.
arXiv:2608. 07346v2 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains.
arXiv:2609.05587v1 Announce Type: new Abstract: Existing evaluations of tool-using agents primarily measure whether an agent can successfully complete diverse tasks with tools. These evaluations gene...
arXiv:2609.24264v1 Announce Type: new Abstract: Tool-use agent traces identify messages and API calls, but procedural analyses also need explicit units of action and inspectable links to their eviden...
arXiv:2608. 07346v1 Announce Type: new Abstract: With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains.
The paper introduces TraceTrial-Bench, a benchmark that evaluates the reliability of AI agent harnesses across seven components—context, execution, governance, lifecycle, observability, tools, and verification—using 95 expert-level tasks with executable or state-based ground truth. It employs a Trajectory Tracing module to record agent execution, a two-layer evaluation framework to separately assess agent performance and harness reliability, and a Multi-source Evidence Acquisition module to gather evidence for LLM-based evaluation. The benchmark’s analysis of 3,420 runs across four harnesses and nine models shows harness scores between 59.8% and 72.0%, with 31.1% of runs succeeding despite failing at least one harness criterion, highlighting the need for auditable and reproducible harness evaluation.