The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
By Radhika Gaonkar
The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.
By Idil Gozel
The study compares four evaluation methods for AI agents in tau2-bench’s airline and retail domains: native scoring against a gold state, an LLM judge, the same judge with a policy document, and a minimal rule-based boundary evaluator. Across 2,160 episodes, the LLM judge agreed with native scoring only 56.2% of the time, while the boundary evaluator detected all 11 forbidden writes that the judges missed. The results suggest that the choice of evaluator significantly influences conclusions about agent correctness and policy compliance.
By Sungbin Im
The paper introduces a new evaluation method called "same-input rerun" to assess the consistency of clinical language‑model agents across repeated runs. By replaying 1,000 MedAgentBench tasks with identical inputs, the authors find that action‑level outputs—such as test orders, medication requests, and referrals—vary significantly, even when benchmark scores remain unchanged. The study demonstrates that current benchmarks, which typically evaluate only a single run per task, can miss substantial behavioral divergence.
By Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han, Wenbin Zhang
The paper investigates whether large language model (LLM) safety judges are truly policy‑invariant, meaning their verdicts should depend only on the agent’s behavior and not on how the evaluation policy is phrased. It introduces three testable principles—rubric‑semantics invariance, rubric‑threshold invariance, and ambiguity‑aware calibration—and applies them in a stress‑test protocol across four agent‑class judges using trajectories from ASSEBench and R‑Judge. The study finds that current judges often change verdicts in response to both meaningful normative shifts and meaningless structural rewrites, with up to 9.1% of verdicts flipping due to content‑preserving rewrites and 18‑43% of flips occurring on unambiguous cases, indicating that safety scores conflate agent actions with evaluator wording.
whyItMatters":"The findings reveal a hidden failure mode in safety evaluation that could mislead researchers about agent performance, underscoring the need for policy‑invariant judges and transparent reporting protocols."
By Shihao Weng, Yang Feng, Xiaofei Xie
LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that...