arXiv AI
Jul 14
AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
arXiv:2607. 11098v1 Announce Type: cross Abstract: Tool-using LLM agents are mostly evaluated assuming all tools work.
Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance
The paper introduces MCP‑Injector, a fault‑injection middleware that simulates realistic failures in the Model Context Protocol used by large‑language‑model agents. Using data from over 57,000 issue and pull‑request threads, the authors calibrate a fault model and run 1,350 experiments across different backbone sizes, agent stacks, and fault conditions. Results show that permanent‑heavy faults drastically reduce task‑completion scores and that larger models (120B) do not gain fault tolerance, highlighting that framework‑level error handling is more critical than model scale.
Related stories
arXiv AI
Aug 18
Evaluating Agentic Code Repair Capabilities in Distributed Systems
arXiv:2608. 14863v1 Announce Type: cross Abstract: LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified.
arXiv AI
Jul 3
A Unified Framework for the Evaluation of LLM Agentic Capabilities
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.
arXiv Machine Learning
Sep 22
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
arXiv:2609.24161v1 Announce Type: cross Abstract: As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet unde...
arXiv AI
Oct 15
When Messages Go Wrong: Understanding Communication Faults in LLM-Based Multi-Agent Systems
arXiv:2610.15459v1 Announce Type: cross Abstract: LLM-based multi-agent systems (LLM-MAS) use specialized agents that exchange messages to accomplish complex tasks. Their reliance on networked protoc...
arXiv AI
Jun 6
A Taxonomy of Runtime Faults in Model Context Protocol Servers
arXiv:2606. 05339v1 Announce Type: cross Abstract: MCP (Model Context Protocol) enables LLMs (Large Language Models) to interact with external tools and data sources via a standardized protocol.
arXiv AI
1d ago
MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents
arXiv:2611.07689v1 Announce Type: new Abstract: Large language model agents are increasingly deployed through the Model Context Protocol (MCP), which allows them to discover and invoke tools across m...
arXiv AI
Jun 2
Self-Healing Agentic Orchestrators for Reliable Tool-Augmented Large Language Model Systems
arXiv:2606. 01416v1 Announce Type: new Abstract: Tool-augmented large language model (LLM) agents rely on orchestration layers that coordinate planning, retrieval, tool invocation, validation, memory, and recovery.
arXiv AI
Jul 17
MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers
arXiv:2607. 14642v1 Announce Type: new Abstract: As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities.
arXiv AI
Sep 3
How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
The paper investigates why large language model (LLM) agents fail on long, multi‑step production workflows despite high benchmark success. By testing nine models (1.2 B–671 B parameters) across six task families and multiple horizons, the authors find that task success follows a geometric decay governed by a per‑step reliability that never reaches 1, leading to inevitable collapse for long horizons. The degradation is driven mainly by step count rather than context length, and the study quantifies a significant gap between benchmark and production performance, especially for agentic tool‑use tasks.
arXiv AI
Jul 7
Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI
arXiv:2607. 02873v1 Announce Type: cross Abstract: Large language model agents driving security tool suites over the Model Context Protocol are increasingly common.
arXiv AI
Jun 6
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents
arXiv:2606. 05806v1 Announce Type: new Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures.