arXiv AI August 11, 2026 By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

Read the original on arXiv AI →

arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Sep 23

SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking

arXiv:2508.15526v2 Announce Type: replace Abstract: The rapid proliferation of large language models (LLMs) has intensified the requirement for reliable safety evaluation to uncover model vulnerabili...

By Xiangyang Zhu, Yuan Tian, Chunyi Li, Kaiwei Zhang, Wei Sun, Guangtao Zhai

llms agents benchmarks safety

arXiv AI

Aug 5

Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

arXiv:2608. 03071v1 Announce Type: new Abstract: Large language model agents derive much of their capability from tool use.

By Guoyao Yu, Xiaoqing Sun, Ziqi Huang, Shaojing Fan, Zhongyi Zhang, Xiaomeng Hu, Xiaobo Xue, Yangyang Shi, Xiong Xiao, Yang Song, Biao Lyu, Rong Wen, Xing Li, Qinming He, Shunming Zhu, Zhenguang Liu

llms rag agents fine-tuning benchmarks

arXiv AI

Aug 12

Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents

arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.

By Bhaskar Gurram

llms agents benchmarks

arXiv AI

Sep 10

DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents

arXiv:2609.06059v1 Announce Type: new Abstract: As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to a...

By Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, Yanbing Liu, Pei Fu

llms agents multimodal benchmarks safety

arXiv AI

Sep 1

APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

arXiv:2608.29128v1 Announce Type: new Abstract: Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matt...

By Zelin Wan, Arash Nourian, Xiaoxiao Li, Nihar Nandan, Kamalakannan Nandagopal

llms agents benchmarks safety

arXiv AI

Jul 28

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.

By Yunting Song, Matthew Watson, Peter Grabowski, Jun Qin

llms reinforcement-learning benchmarks safety data-engineering

One email a morning, machine-written

Related stories

SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking

Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents

DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents

APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation