arXiv AI
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 22
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
arXiv:2609.24161v1 Announce Type: cross Abstract: As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet unde...
arXiv AI
Jul 14
AgentAbstain: Do LLM Agents Know When Not to Act?
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
arXiv AI
Jun 16
ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
arXiv:2606. 15508v1 Announce Type: new Abstract: Tool-augmented large language model agents increasingly operate over large tool libraries, but existing evaluations often focus on whether a model can call a tool correctly rather than how the visible tool menu shapes reliability, efficiency, and safety-relevant risk exposure.
arXiv Machine Learning
Oct 19
Tool Recommendation for Multi-Step Agents: Finding Every Tool a Task Needs
arXiv:2610.23707v1 Announce Type: new Abstract: A multi-step agent task fails when a single tool it needs is missing, yet the tools an agent may need are too many to list: the roughly 8,200 tools on...
arXiv AI
Oct 19
Explore Separately, Adopt Selectively: Execution-Guided Tool Use over Large Tool Libraries
The paper introduces BRIA (BRanch-Inspect-Adopt), an agent that isolates tool‑candidate investigations into separate branches, executing each candidate independently and then inspecting the execution evidence to adopt only useful results. BRIA dynamically retrieves additional tools as new needs arise during task execution and is evaluated on PlanBench‑XL and MCP‑Atlas with a library of 1,897 tools, outperforming or matching the strongest retrieval baselines. On MCP‑Atlas, BRIA achieves an average claim score of 89.77 versus 75.40 for MCP‑Zero, and it lies on the Pareto frontier for task performance and input‑token cost.
arXiv AI
Jul 17
MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers
arXiv:2607. 14642v1 Announce Type: new Abstract: As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities.