The paper investigates whether locally deployed large language models can automate hardware design workflows that involve repetitive, dependency-ordered operations using specialized tools. A Model Context Protocol (MCP) server is created to emulate a proprietary hardware design tool, and a benchmark tests single and multi-step edits, invalid requests, misspelled prompts, and multi-server contexts. Seven open-source models are evaluated across different pipeline choices, revealing that strong models can nearly fully cover expected calls, but reliability hinges on task structure and agent configuration, with comprehensive tool descriptions reducing failures and multi-agent setups aiding weaker models at the cost of extra calls.
arXiv:2607. 14642v1 Announce Type: new Abstract: As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities.
Octopus Protocol is a hardware onboarding framework that allows an AI coding agent to automatically discover, identify, and integrate hardware devices into an AI system. Using a single bootstrap command, the agent runs a five-stage pipeline to enumerate visible hardware, infer device capabilities, generate typed Model Context Protocol tools, produce the necessary code, and activate a live endpoint. The system maintains a persistent daemon that repairs deployment failures, enabling consistent, platform‑agnostic interfaces across diverse hosts without manual integration code.
arXiv:2607. 11138v1 Announce Type: new Abstract: The rapid expansion of capabilities in Large Language Model (LLM) agents has exposed a critical architectural bottleneck: when agents are given access to a flat, monolithic registry of tools, the model must evaluate hundreds or thousands of options simultaneously.
arXiv:2605. 10907v3 Announce Type: replace-cross Abstract: The dominant paradigm for AI agents is an "on-the-fly" loop in which agents synthesize plans and execute actions within seconds or minutes in response to user prompts.
arXiv:2607. 23933v1 Announce Type: cross Abstract: As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency.
Agent Seer is a pipeline that automatically synthesizes realistic evaluation scenarios for AI agents that use external tools, using only the tool’s specification (function names, natural‑language descriptions, and typed parameter schemas). Starting from a single Model Context Protocol (MCP) specification, it enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock‑data‑grounded multi‑turn dialogues that demonstrate strong tool‑calling correctness and conversational coherence. Across seven diverse MCP specifications, the pipeline achieves high quality, with parameter‑schema complexity emerging as the main driver of quality variation and argument‑value accuracy identified as the dominant failure mode.
The paper surveys how large language models (LLMs) are being used in hardware verification, covering tasks such as assertion generation, testbench creation, bug localization, and formal reasoning. It categorizes the literature by methodology, verification goal, tool interaction, benchmarks, and evaluation metrics, and distinguishes between inference-time techniques (prompting, retrieval, structured reasoning, agentic workflows) and training-time adaptation. The authors highlight that LLMs excel as semantic reasoning, search, and orchestration components within verification workflows, yet emphasize the need for semantic alignment, scalable integration, generalization, and rigorous evaluation to ensure true verification correctness.
GatewayBench is a new evaluation framework that systematically studies how the design of Model Context Protocol (MCP) gateways influences AI agent capability, efficiency, and security. It categorizes gateways into flattened and hierarchical architectures and tests them on 524 tasks across six domains using a pool of 11,118 real-world tools from 251 MCP servers, including task-irrelevant tools to simulate large-scale deployments. Experiments on seven widely deployed gateways reveal significant differences in task completion, tool-use efficiency, and attack resilience, showing strong interactions with the underlying model and agent harness.
arXiv:2609.24161v1 Announce Type: cross Abstract: As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet unde...
QiMeng-ChipV-RTL is a multi‑agent framework that tackles the challenges of generating Register‑Transfer Level (RTL) code for industrial IP‑level hardware design. By partitioning long design documents into short, localized tasks and using hierarchical planning, localized code generation, interface‑consistent merging, and AST‑guided debugging, it scales to complex specifications. Experiments on the RealBench benchmark show ChipV-RTL achieves a 45.0% pass rate, outperforming state‑of‑the‑art LLMs and agents which reach only 21.6%.
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.
arXiv:2606. 14350v1 Announce Type: cross Abstract: Artificial Intelligence (AI) systems must typically satisfy service-level objectives including accuracy, latency, and cost.