ToolRobustBench is a stage-wise diagnostic benchmark designed to evaluate and diagnose failures in tool‑calling agents, which are large language models that select tools, provide structured arguments, and interpret tool feedback. The benchmark aligns four perturbation families—tool‑interface, user‑intent, tool‑output/observation, and runtime‑environment—with the tool‑use pipeline, attributing failures to specific stages such as tool selection, schema grounding, argument binding, and feedback handling. Experiments across 15,456 instances, 7 models, and 16 local tools reveal that while overall performance is high, robustness degrades significantly, especially under tool‑output/observation perturbations, and mixed‑family perturbations produce non‑additive failure patterns.
By YiShan Zheng, Yuan Wu, Yi Chang
arXiv:2608. 02645v1 Announce Type: cross Abstract: Large Language Model (LLM) agents rely on external tools to perform multistage tasks.
By Isham Kalappurackal Mansoor, Abhishek Phadke, Pratip Rana
arXiv:2607. 11098v1 Announce Type: cross Abstract: Tool-using LLM agents are mostly evaluated assuming all tools work.
By Aritra Mazumder, Nusrat jahan Lia
arXiv:2606. 01416v1 Announce Type: new Abstract: Tool-augmented large language model (LLM) agents rely on orchestration layers that coordinate planning, retrieval, tool invocation, validation, memory, and recovery.
By Rahul Suresh Babu, Adarsh Agrawal
Bench2Robust is a framework that augments existing tool‑using LLM benchmarks with scenario‑controlled solvability, adding episodes that require retrying the original path, switching to a fallback tool, or recognizing that no viable path remains. It separates recovery‑policy quality from randomness in failure persistence, and across 7 models and 10 task slices, 69 of 70 model‑subset pairs degrade under tool failures by up to 46.7 percentage points. The authors introduce Bayesian Tool Memory (BTM) for structured runtime recovery context and use reinforcement learning (RL) for learned recovery behavior, achieving up to a 20.7‑point improvement over the base model without sacrificing clean performance.
By Chaoran Chen, Vy Nguyen, Ziji Zhang, Abhinav Gullapalli, Ziyi Wang, Yuxuan Lu, Dakuo Wang, Jing Huang, Zhou Yu, Jin Lai
arXiv:2607. 04686v1 Announce Type: cross Abstract: Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails.
By Harsh Soni
arXiv:2608. 15579v1 Announce Type: cross Abstract: Industrial software-engineering teams increasingly need LLM agents that turn bug reports into correct patches, yet benchmark-scale operation adds long horizons, tool-use discipline, context persistence, heterogeneous clusters, and evaluation reuse.
By Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, Satoshi Nakashima, Yu Ishikawa, Kosuke Maeda, Nao Soma, Kenichi Kobayashi, Keisuke Miyazaki, Keizo Kato, Shigeki Fukuta, Tatsuo Kumano, Nobutaka Imamura, Kevin Musgrave, Shahbaz Abdul Khader, Kwun Ho Ngan, Joe Townsend, Fayas Asharindavida, Matthieu Parizy, Akira Sakai, Yuma Ichikawa, Yang Zhao, Michiaki Takizawa, Taku Fukui, Hiroki Ohtsuji, Wei-Peng Chen, Hiromichi Kobashi
The paper investigates how the representation of tool interfaces—referred to as a tool schema—affects the performance of large language model agents. By systematically transforming a native tool schema with nine operators, the authors demonstrate that agents exhibit significant variability in success rates across different but functionally equivalent schema variants, a phenomenon they call schema bias. Experiments on eleven LLMs, including closed models, reveal that success can range from complete failure to 97% solely due to schema differences, and that training only mitigates bias when the specific variant is included in the training data.
By Yinhong Liu, Zhili Tan, Zilin Wang, Zhijiang Guo
RubricRefine is a training‑free pre‑execution refinement method that generates task‑specific rubrics from tool documentation, scores candidate code against explicit contract checks, and iteratively repairs failures before execution. It achieves an average score of 0.86 across seven models on M3ToolEval without any execution attempts, outperforming prior inference‑time baselines while incurring lower latency. The approach shows consistent performance on single‑step API‑Bank tasks and maintains an advantage in multi‑turn settings on AppWorld, with its effectiveness tied to the quality of the supplied documentation.
By Will LeVine, Brendan Evers, Sam Saltwick, Abhay Venkatesh
arXiv:2606. 05806v1 Announce Type: new Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures.
By Dongsheng Zhu, Xuchen Ma, Yucheng Shen, Xiang Li, Yukun Zhao, Shuaiqiang Wang, Lingyong Yan, Dawei Yin
The paper introduces Growing Harness, a training method that transforms recurring control logic in large language model agents into reusable executable code, reducing reliance on the model for task-specific decisions. By using strategy-free scaffolds, failure-guided code repair, and success-first gating, the approach learns a shared harness that improves performance across multiple benchmarks and model sizes. Experiments on BrowseComp-Plus and WebArena-Verified show significant gains in success rates and substantial reductions in LLM calls and inference cost compared to traditional tool‑calling agents.
By Laizhen Li, Jiarui Li, Juanjuan Zhao, Kejiang Ye, Ye Li, Cheng-zhong Xu, Xitong Gao
arXiv:2610.16333v1 Announce Type: new
Abstract: As AI agents increasingly act through external tools, tool results are no longer simple outputs but execution artifacts that must remain correctly asso...
By Yutong Zhi, Jie Zhang, Xiuqi Lu, Jiayuan Xu, Jing Shao