arXiv AI
Jul 14
AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
arXiv:2607. 11098v1 Announce Type: cross Abstract: Tool-using LLM agents are mostly evaluated assuming all tools work.
ProbeID: Verifying MCP Tool Identity for LLM Agents under Registry Drift
Related stories
arXiv AI
Sep 30
Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments
arXiv:2609.37315v1 Announce Type: cross Abstract: Tool-using agents are entering settings where a wrong action carries real cost, and the benchmarks certifying them grade what each simulated tool cal...
arXiv AI
Aug 12
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv AI
Sep 12
What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
The study examines the composition of a random sample from the Model Context Protocol (MCP) registry, revealing that only 48.8% of the 400 sampled npm/stdio servers successfully complete an initialization handshake, compared to 66.7% for a hand‑curated frame. Among the servers that run, there are no fatal JSON Schema violations across 2,766 advertised tools, but optional safety annotations vary widely, with a 58.8% omission rate in the random draw versus 41.5% in the curated set. The authors also compare MCP tool descriptions to two benchmark corpora, finding minimal near‑duplication in real MCP tools (2.8%) and significant repetition in synthetic datasets (up to 85.6%).
arXiv AI
Sep 1
The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents
arXiv:2608.28795v1 Announce Type: cross Abstract: Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screensho...
arXiv AI
Sep 17
AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
AutoTuneBench introduces a trustworthy measurement protocol for evaluating how large language model agents auto‑tune GPU kernels and serving engines. The benchmark addresses four failure modes—strawman baselines, machine‑dependent timing, saturated tasks, and infrastructure defects—by enforcing code‑frozen protocols, database validation, anti‑cheat checks, pre‑registered comparisons, and external result anchoring. Using this protocol, the authors demonstrate that previously reported speedups are inflated, revealing more modest improvements across different engines and machines.
arXiv AI
Oct 5
Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...
arXiv Machine Learning
5d ago
What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer
The paper investigates how standard evaluations of detectors for hijacked large language model (LLM) agents may conceal important weaknesses. By auditing eleven evaluations on the AgentDrift and AgentDojo benchmarks, the authors show that detectors often perform poorly when faced with rare attacks, unseen tools, or data from a different benchmark, with precision dropping dramatically at low attack rates and rankings reversing as tool coverage falls. They also find that a released LLM action monitor, despite high AUROC on its own data, would still generate many false alerts when applied to a new benchmark. "whyItMatters":"The study highlights that current evaluation practices can give a misleading sense of detector reliability, underscoring the need for more robust, cross‑benchmark testing to ensure real‑world safety of LLM agents."
arXiv AI
Oct 23
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
The paper introduces VerifierBench, a benchmark comprising 222 tasks from six agent benchmarks and 4,993 hidden test cases designed to evaluate the generation of executable verifiers by large language model agents. It presents Verifier4Verifier (V4V), a six-check audit that screens source verifiers and test suites, and reports that across 19 models, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers more reliably rejecting invalid solutions than accepting valid ones. Case studies highlight that many false rejections stem from verifiers that accept no tested solution, revealing unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions, and underscore the need to test both acceptance of correct solutions and other permissible solutions.
arXiv AI
Oct 1
Hard-Gate Candidacy in a Deployed Validator Suite
The paper evaluates hard‑gate candidacy for validators in a deployed generative‑agent system by measuring how well each validator’s firing separates successful from failed builds. Across 13 validators and thousands of builds, only a few checks show statistically significant separation, while many fail to distinguish or never fire. The study highlights that skipped checks are recorded as passes, limiting detectable failure rates and underscoring the need for clearer evaluation records.
Hugging Face Trending Papers
Sep 10
What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines t...
arXiv Machine Learning
Sep 30
Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.