arXiv AI
Jul 14
AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
arXiv:2607. 11098v1 Announce Type: cross Abstract: Tool-using LLM agents are mostly evaluated assuming all tools work.
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
AgentProofBench is a benchmark for reusable verification of web agents, linking frozen acceptance rules to execution from identical saved openings. It covers 97 tasks across 79 families and four web services, and provides paired tests that measure how well programs retain correctness after reuse and rebuild. Experiments with DeepSeek Flash and Pro show high retention rates and identify specific families where deployment-state panel correctness is critical.
Related stories
arXiv AI
Jul 1
An Executable Benchmarking Suite for Tool-Using Agents
arXiv:2605. 11030v2 Announce Type: replace-cross Abstract: Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims.
arXiv AI
Oct 23
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
The paper introduces VerifierBench, a benchmark comprising 222 tasks from six agent benchmarks and 4,993 hidden test cases designed to evaluate the generation of executable verifiers by large language model agents. It presents Verifier4Verifier (V4V), a six-check audit that screens source verifiers and test suites, and reports that across 19 models, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers more reliably rejecting invalid solutions than accepting valid ones. Case studies highlight that many false rejections stem from verifiers that accept no tested solution, revealing unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions, and underscore the need to test both acceptance of correct solutions and other permissible solutions.
arXiv AI
Oct 9
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.
arXiv AI
Jul 3
A Unified Framework for the Evaluation of LLM Agentic Capabilities
arXiv:2605. 27898v2 Announce Type: replace Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential.
arXiv AI
Jun 9
SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows
arXiv:2606. 08049v1 Announce Type: new Abstract: AI agents increasingly turn past experience into reusable artifacts such as code, workflows, and procedural memories.
arXiv AI
3d ago
SkillLifeBench: Benchmarking Security Across the Lifecycle of Agent Skills
arXiv:2610.35215v1 Announce Type: cross Abstract: Web agents reuse skills whose stored code, selected version, and loaded implementation can diverge across tasks and updates. SkillLifeBench links the...
arXiv AI
Sep 30
LatentSift: Policy-State Filtering for Token-Efficient Verification of Software Engineering Agents
LatentSift is a token‑free, execution‑free filtering method for software engineering agents that replaces the initial LLM‑based verifier. It represents each candidate trajectory using the policy’s hidden states—reasoning, observation, and function‑call states—and compares them against banks of successful and unsuccessful states collected during training. By fusing distance scores with a learned linear score, LatentSift retains promising candidates, reducing EF‑verifier tokens by 66.6–81.0% and total verification tokens by 49.1–62.1% while maintaining or improving performance on DeepSWE‑Preview.
arXiv Computation and Language
Aug 31
RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
RealClawBench is a live benchmark framework derived from real OpenClaw developer‑agent sessions, designed to capture the distribution, diversity, and real‑world difficulty of deployed agent use. It reconstructs execution environments and uses deterministic verifiable scorers to convert real sessions into reproducible, automatically scored tasks, yielding 281 executable tasks with minimal distribution shift. Evaluation of 14 contemporary models shows the best system solves only 65.8% of tasks, highlighting significant room for improvement on realistic workloads.
arXiv AI
Sep 1
The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents
arXiv:2608.28795v1 Announce Type: cross Abstract: Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screensho...
arXiv AI
Oct 16
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
Harness-Bench is a diagnostic benchmark that evaluates how different harness configurations affect LLM agent performance in realistic workflows. It tests 106 sandboxed offline tasks across multiple model backends, recording artifacts, execution traces, and usage statistics to analyze completion, process quality, efficiency, and failure behavior. The study finds significant variation in outcomes across model-harness pairings, indicating that agent capability should be reported at the configuration level rather than solely by the base model.
arXiv AI
Jun 9
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
arXiv:2606. 07682v1 Announce Type: cross Abstract: AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments.