arXiv AI
Jul 7
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
arXiv:2607. 04686v1 Announce Type: cross Abstract: Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails.
ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets
ToolBottleneckBench is a benchmark that evaluates tool‑agent performance under controlled pressures and fixed harness budgets. It keeps synthetic enterprise tasks constant while varying one pressure—such as the registered tool catalog, visible tool menu, certified workflow, or record encoding—to measure how each affects success. Across 21 model and harness configurations, the visible menu pressure uniquely imposes a large penalty on the model, largely due to truncation at the completion cap, and most failures are driven by harness limits rather than tool‑specific issues.
Related stories
arXiv AI
Oct 16
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
Harness-Bench is a diagnostic benchmark that evaluates how different harness configurations affect LLM agent performance in realistic workflows. It tests 106 sandboxed offline tasks across multiple model backends, recording artifacts, execution traces, and usage statistics to analyze completion, process quality, efficiency, and failure behavior. The study finds significant variation in outcomes across model-harness pairings, indicating that agent capability should be reported at the configuration level rather than solely by the base model.
arXiv AI
Aug 28
Same Model, Different Harness: Different Coding-Agent Results
The paper investigates how altering the harness—specifically the way a coding agent manages context and tool outputs—affects performance when the underlying model and task remain unchanged. Two harness configurations were compared on three coding benchmarks: a control that preserves the full conversation in order, and a treatment that mechanically shortens older tool results to keep the context tight. Across all benchmarks, the treatment increased the mean per‑task fail‑to‑pass fraction and, in some cases, the number of complete solutions, demonstrating that the harness itself can significantly influence a frozen model’s effectiveness.
arXiv AI
Oct 2
Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks
Mingbird is a local‑first agent harness designed for small open‑weight language models (2–9 B) that run on ordinary laptops. It introduces ten mechanisms—such as a byte‑level net‑zero prefill budget, a finish gate that re‑reads the task before accepting completion, and signature‑level loop detection—to address common failure modes that arise from the harness rather than the model itself. In controlled experiments on the LRAB benchmark and the $ au^2$‑bench, Mingbird achieves higher overall scores (0.886 and 0.856 respectively) compared to other harnesses, and its ablation studies show that each mechanism contributes measurable performance gains.
arXiv AI
Aug 26
ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
ToolRobustBench is a stage-wise diagnostic benchmark designed to evaluate and diagnose failures in tool‑calling agents, which are large language models that select tools, provide structured arguments, and interpret tool feedback. The benchmark aligns four perturbation families—tool‑interface, user‑intent, tool‑output/observation, and runtime‑environment—with the tool‑use pipeline, attributing failures to specific stages such as tool selection, schema grounding, argument binding, and feedback handling. Experiments across 15,456 instances, 7 models, and 16 local tools reveal that while overall performance is high, robustness degrades significantly, especially under tool‑output/observation perturbations, and mixed‑family perturbations produce non‑additive failure patterns.
arXiv AI
Oct 1
Evaluating Agents Across Runtime Contracts: When Mismatch Costs Efficiency or Quality
arXiv:2603.01209v3 Announce Type: replace Abstract: In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python v...
arXiv AI
Sep 30
Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments
arXiv:2609.37315v1 Announce Type: cross Abstract: Tool-using agents are entering settings where a wrong action carries real cost, and the benchmarks certifying them grade what each simulated tool cal...
arXiv AI
Jun 16
ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
arXiv:2606. 15508v1 Announce Type: new Abstract: Tool-augmented large language model agents increasingly operate over large tool libraries, but existing evaluations often focus on whether a model can call a tool correctly rather than how the visible tool menu shapes reliability, efficiency, and safety-relevant risk exposure.
arXiv AI
1d ago
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
arXiv:2611.08417v1 Announce Type: new Abstract: Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even be...
arXiv AI
Sep 10
Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses
arXiv:2609.05736v2 Announce Type: new Abstract: LLM tool agents can be improved without retraining by modifying the runtime harness around a fixed model: prompts, tool interfaces, middleware, state h...
Hugging Face Trending Papers
Oct 14
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the...
Hugging Face Trending Papers
2d ago
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release...