The paper critiques current GUI grounding benchmarks that count a click as correct only when it falls inside a single annotated box, arguing that many interfaces expose the same action through multiple elements. The authors propose redefining correctness as achieving the same outcome as clicking the annotated target, and introduce a probe that executes and fingerprints thousands of clicks per page to create outcome maps and an execution judge. Applying this method to several large‑scale grounders on real web applications shows that a significant fraction of clicks rejected by box‑based scoring actually produce the intended outcome, indicating that box scoring under‑counts correct clicks and that evaluation metrics need adjustment.
arXiv:2610.17747v1 Announce Type: cross Abstract: Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step agai...
arXiv:2607. 05904v1 Announce Type: new Abstract: Training a language model against its own reference-free judgments (the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines) assumes a model's verdict on a shown answer tracks correctness.
StrictPRMBench is a replayable evaluation benchmark designed to audit fixed‑score PRM aggregation rules. It standardizes mathematical reasoning pools, scorer evidence, and deletion interventions so that different reducers confront identical selection problems. Across seven checkpoints and two benchmarks, the benchmark reveals that product aggregation yields higher deletion‑induced selection instability than geometric mean, with statistically significant differences in most cells.
arXiv:2605. 30000v2 Announce Type: replace Abstract: Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale.
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.
arXiv:2606. 16062v1 Announce Type: new Abstract: We measure the rate at which code RL environments accept incorrect solutions as correct.
arXiv:2610.17895v2 Announce Type: cross Abstract: Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess...
arXiv:2609.24890v1 Announce Type: cross Abstract: Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with f...
arXiv:2607. 11598v1 Announce Type: new Abstract: There are two standard ways to spend more compute at test time: let a model reason longer, or sample more attempts and keep one.
arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input...
arXiv:2610.00111v1 Announce Type: new Abstract: Model judges now supervise multimodal systems at scale, filtering training data, selecting outputs, and supplying the reward that shapes multimodal rea...