arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
By Bhaskar Gurram
arXiv:2609.14758v1 Announce Type: cross
Abstract: Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply...
By Arham Sethi, Arsen Kenzhebayev, Saanvi Paturi, Vatsal Raina, Vyas Raina, Ivaxi Sheth
arXiv:2607. 19449v1 Announce Type: cross Abstract: Evaluation frameworks for tool-augmented LLM agents focus overwhelmingly on capability metrics or explicit tool crashes, leaving silent infrastructure failures and HTTP 200 responses with empty, null, or malformed payloads largely unaudited.
By Aarushi Singh
arXiv:2607. 20436v1 Announce Type: cross Abstract: Safety evaluations often assume that behavior observed during testing reflects behavior in ordinary use, but fine-tuning can break this assumption.
By Phongsakon Mark Konrad, Toygar Tanyel, Serkan Ayvaz
The paper argues that accuracy is an inadequate metric for certifying a coordinator’s theory‑of‑mind in multi‑agent systems, because the required accuracy depends on deployment‑specific costs and errors are often dependent. Experiments on task assignment, Level‑Based Foraging, Hanabi, and human judgments show that accuracy can both approve harmful language‑model deployments and reject beneficial ones. The authors propose a Blind‑Spot Gate that evaluates error costs, pairwise interactions, and dependence without deployment, achieving better safety outcomes than accuracy alone.
By Will Beaumaster, Paul Schrater
The paper introduces CORA (Counterfactual, Observable Redundancy Audit), a protocol for auditing website redundancy by measuring repetition load, normal-use tax, and failure-domain recovery reserve. Each audit run records screenshots, stable element identities, and task traces, while a versioned vision‑language model generates annotations that are validated and released only if they meet calibrated criteria. Experiments on a transparent mechanistic testbed show that CORA’s factorized representation separates reserve from normal-use tax and predicts perturbed success more accurately than scalar-load baselines, but it withholds automated scores when instruments fail to meet release requirements, indicating that CORA is an auditable candidate procedure for the studied benchmark rather than a universal standard.
By Ge Kong, Yongtong Cao