arXiv Machine Learning
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Sep 15
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
arXiv:2512.24565v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol...
arXiv AI
Jul 17
MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers
arXiv:2607. 14642v1 Announce Type: new Abstract: As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities.
arXiv AI
Aug 18
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.
arXiv AI
Aug 26
Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation
Hybrid Semantic Tool Discovery for Enterprise MCP Gateway presents SCOUT, a system that addresses two major challenges in large language model (LLM) agent tool usage: a context‑engineering bottleneck and a tool discoverability barrier. SCOUT reframes tool exposure as a context‑selection problem, injecting only relevant tools into the model’s context window and providing two MCP meta‑tools—tool_search and execute_tool—to perform hybrid retrieval via BM25 and dense vector search. In production at PayPal, SCOUT cuts MCP tool‑token consumption by 99%, dramatically reducing per‑query inference cost while remaining model‑agnostic and requiring no client‑side changes.
arXiv Machine Learning
Jul 14
ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory
arXiv:2607. 11126v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly rely on external tools served by shared providers and accessed by heterogeneous downstream agents.
arXiv AI
Sep 2
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.