arXiv:2607. 04686v1 Announce Type: cross Abstract: Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails.
By Harsh Soni
Harness-Bench is a diagnostic benchmark that evaluates how different harness configurations affect LLM agent performance in realistic workflows. It tests 106 sandboxed offline tasks across multiple model backends, recording artifacts, execution traces, and usage statistics to analyze completion, process quality, efficiency, and failure behavior. The study finds significant variation in outcomes across model-harness pairings, indicating that agent capability should be reported at the configuration level rather than solely by the base model.
By Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Zongwei Lv, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Guangxiang Zhao, Tong Yang
The paper investigates how altering the harness—specifically the way a coding agent manages context and tool outputs—affects performance when the underlying model and task remain unchanged. Two harness configurations were compared on three coding benchmarks: a control that preserves the full conversation in order, and a treatment that mechanically shortens older tool results to keep the context tight. Across all benchmarks, the treatment increased the mean per‑task fail‑to‑pass fraction and, in some cases, the number of complete solutions, demonstrating that the harness itself can significantly influence a frozen model’s effectiveness.
By Sydney Lewis
Mingbird is a local‑first agent harness designed for small open‑weight language models (2–9 B) that run on ordinary laptops. It introduces ten mechanisms—such as a byte‑level net‑zero prefill budget, a finish gate that re‑reads the task before accepting completion, and signature‑level loop detection—to address common failure modes that arise from the harness rather than the model itself. In controlled experiments on the LRAB benchmark and the $ au^2$‑bench, Mingbird achieves higher overall scores (0.886 and 0.856 respectively) compared to other harnesses, and its ablation studies show that each mechanism contributes measurable performance gains.
By Hao Wang, Ting Huang
ToolRobustBench is a stage-wise diagnostic benchmark designed to evaluate and diagnose failures in tool‑calling agents, which are large language models that select tools, provide structured arguments, and interpret tool feedback. The benchmark aligns four perturbation families—tool‑interface, user‑intent, tool‑output/observation, and runtime‑environment—with the tool‑use pipeline, attributing failures to specific stages such as tool selection, schema grounding, argument binding, and feedback handling. Experiments across 15,456 instances, 7 models, and 16 local tools reveal that while overall performance is high, robustness degrades significantly, especially under tool‑output/observation perturbations, and mixed‑family perturbations produce non‑additive failure patterns.
By YiShan Zheng, Yuan Wu, Yi Chang
arXiv:2603.01209v3 Announce Type: replace
Abstract: In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python v...
By Victor May, Van Khue Nguyen, Aaditya Salgarkar, Yishan Wang, Diganta Misra, Huu Nguyen