arXiv AI

ReA-Bench: Benchmarking Responsible Agent Auditing

ReA-Bench is a diagnostic benchmark designed to evaluate how well auditors detect risks in autonomous agent trajectories. It categorizes auditing challenges into six tiers—internal safety knowledge, contradictions, evidence retrieval, source verification, indirect steering, and missing information—and includes both realistic production samples and challenging benign cases. The benchmark also provides paired trajectory versions to assess the impact of trajectory length and information availability on risk detection reliability.

arXiv AI
Aug 14

Auditable Agents

arXiv:2604. 05485v2 Announce Type: replace Abstract: LLM agents call tools, query databases, delegate tasks, and trigger external side effects.

By Yi Nian, Aojie Yuan, Haiyue Zhang, Jiate Li, Li Li, Xiyang Hu, Hua Wei, Xiongye Xiao, Chaowei Xiao, Yue Zhao
arXiv AI
Oct 16

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

PROBE is an algorithm that trains language‑model auditors to probe deployed tool‑using agents for policy violations through realistic, multi‑turn, tool‑mediated dialogue. By framing auditing as a two‑player Markov game and using a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity, PROBE outperforms zero‑shot adversaries in uncovering broader and higher‑yield policy violations. The study compares GRPO, DPO, and GFlowNet optimizations, demonstrating that multi‑turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing existing capability and safety benchmarks.

By Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou
arXiv AI
6d ago

Knowing When to Intervene: Causal Evaluation of AI Agents

The paper introduces a framework for assessing the causal impact of interventions on AI agents during deployment, moving beyond static benchmarks. By collecting a modest set of agent trajectories under randomized interventions such as verification or screening, the authors estimate dynamic causal effects for various contextual intervention policies. These estimates enable the creation of safety leaderboards that rank agent actions by risk, aiding audit prioritization across benchmarks like tau‑bench Retail, ToolSandbox, and ImpossibleBench.

By Ayush Sawarni, Wenqian Xing, Tijana Zrnic, Ramesh Johari, Vasilis Syrgkanis
arXiv AI
Aug 7

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.

By Zhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
arXiv AI
Oct 5

Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

The paper introduces BenchGuard, a framework that uses advanced language models to audit execution-based agent benchmarks. BenchGuard systematically cross-verifies benchmark artifacts through structured LLM protocols, optionally incorporating agent solutions or execution traces for deeper diagnostics. In tests on ScienceAgentBench and BIXBench Verified-50, BenchGuard uncovered numerous author-confirmed and expert-identified issues, including fatal errors that made tasks unsolvable, and demonstrated cost-effective auditing for complex bioinformatics tasks.

By Xinming Tu (Minta), Tianze Wang (Minta), Yingzhou (Minta), Lu, Kexin Huang, Yuanhao Qu, Sara Mostafavi
arXiv AI
Oct 19

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

AuditKIT is a source‑available evaluation library that unifies benchmark accuracy, judge‑based assessment, runtime performance, tool‑use behavior, and retrieval quality into a single result representation. It offers direction‑aware metric contracts, configuration fingerprints, and extensible metrics and backends, while providing turn‑aware tool‑call matching and distinguishing verified task outcomes from trajectory‑level diagnostics. Seven case studies demonstrate how answer extraction, runtime behavior, and metric selection can alter evaluation outcomes across model comparison, serving runtimes, retrieval, tool calling, and interoperability.

By Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth