arXiv AI

When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

The study compares four evaluation methods for AI agents in tau2-bench’s airline and retail domains: native scoring against a gold state, an LLM judge, the same judge with a policy document, and a minimal rule-based boundary evaluator. Across 2,160 episodes, the LLM judge agreed with native scoring only 56.2% of the time, while the boundary evaluator detected all 11 forbidden writes that the judges missed. The results suggest that the choice of evaluator significantly influences conclusions about agent correctness and policy compliance.

arXiv Machine Learning
Sep 30

Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.

By Jiapeng Li
arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv AI
Sep 3

LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails

The paper argues that using a large language model (LLM) as the sole judge in self‑improving agent pipelines is problematic, as the judge can be biased or manipulated, leading to false confidence in system performance. The authors propose a new framework, PROCTOR, which replaces the oracle judge with a deterministic, teacher‑student loop that enforces guardrails such as sandboxing, role separation, and acceptance checks to prevent cheating and ensure reliable evaluation. Experiments across contract analysis, compliance review, and code quality demonstrate that PROCTOR mitigates eleven identified failure modes that previously allowed agents to achieve perfect scores while hiding significant capability gaps.

By Vansh Wahi
arXiv Computation and Language
Oct 9

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.

By Radhika Gaonkar
arXiv AI
Sep 11

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

AgentAudit is an open, extensible framework that evaluates the full lifecycle of AI agents, assessing planning, tool selection, execution, memory, and reasoning across ten dimensions such as instruction integrity, security, and alignment. Unlike existing benchmarks that focus on single aspects, AgentAudit analyzes the entire execution trace to attribute failures to specific stages. The framework was tested on five large language models, revealing significant differences in trustworthiness even among models with similar task‑completion performance.

By Shrey Nag, Sachita, Abhishek Kumar Singh, Lipi Goel, Rajeshwar Singh Janwar
arXiv Computation and Language
Oct 23

From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

The paper presents a clinician‑led safety guardrail for mental‑health chatbots that replaces typical prompt‑engineering or Constitutional AI methods. Two clinicians design, finalize, and adjudicate a rule base of four safety checks, which is formally verified for internal consistency and applied to LLM outputs. The verification layer’s verdicts are recomputable from a named rule and sixteen facts, and it consistently avoids false crisis‑resource demands that a matched LLM judge frequently raises.

By Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani