arXiv Computer Vision

A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversight

arXiv AI
Nov 5

Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge

The paper investigates confirmation bias in Agent-as-a-Judge systems, which evaluate open-ended agent tasks by gathering evidence from the agent’s trajectory. It demonstrates that a single false pre‑screening note can overturn up to 40% of correct verdicts across multiple judges and datasets, even when the judge collects decisive evidence. The authors classify the resulting errors into over‑trust, hallucination, and under‑investigation, noting that over‑trust dominates in larger Claude judges, and show that lightweight mitigations reduce but do not eliminate the bias.

By Drake Benjamin Brown, Khushbu Pahwa, Yubin Ge, Salvatore Romeo, Akhil Agnihotri, Yi Zhang, Xiaofei Ma, Daniele Bonadiman
arXiv Computer Vision
Nov 9

Grounding-Aware Judge Reward Reinforcement Learning for Out-of-Domain Face Anti-Spoofing

The paper proposes a grounding‑aware judge reward for reinforcement learning to improve out‑of‑domain face anti‑spoofing. A 32B vision‑language judge evaluates each reasoning chain for evidence anchoring, logical coherence, and domain knowledge, providing a verification signal that replaces sparse verdict‑only rewards. Experiments show the reward reduces ACER on two physical benchmarks while maintaining in‑domain accuracy, outperforming several baselines.

By Xiaoyong Yu, Xinge Yu
arXiv Machine Learning
Sep 25

Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

The paper investigates how providing execution traces to multimodal judges in agentic video‑generation systems can bias their verdicts. On a benchmark of 109 two‑event clips, traces that falsely report successful tool calls cause large‑language‑model judges to incorrectly accept 78–90 % of failures, while contradictory traces lead to 100 % rejection of correct clips. The effect persists even when judges are instructed to consider only the video frames, indicating that the vulnerability stems from the judges’ learned trust in tool logs rather than the visual content itself.

By Jian Xu
arXiv Computer Vision
Oct 12

Auditing VLM Benchmarks from Existing Evaluation Runs

The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.

By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov
arXiv AI
Oct 27

Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models

Vision–language models can correctly diagnose medical questions using only the clinical vignette, without the diagnostic image. In experiments on 60 Italian State Examination items, models defended diagnoses derived from a blank placeholder as firmly as those derived from real images, showing no increased fragility. While different adversarial pressures produced distinct dynamics—one leading to a single decisive flip and the other to continual changes—the instability under contentless doubt reflected item ambiguity rather than lack of evidence.

By Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti, Olivia Riccomi, Michele Ferramola, Vittorio De Vita, Alessandro Tosi, Antonio Cristiano, Alessia Longo, Lorenzo De Mori, Chiara Battipaglia, Melissa Sawaya, Luigi De Angelis, Marcello Di Pumpo, Alessandra Piscitelli, Pietro Eric Risuleo, Giulia Vojvodic, Mariapia Vassalli, Nicolo Scarsi, Manuel Del Medico
arXiv AI
Nov 6

Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test

The paper investigates whether large language model (LLM) safety judges are truly policy‑invariant, meaning their verdicts should depend only on the agent’s behavior and not on how the evaluation policy is phrased. It introduces three testable principles—rubric‑semantics invariance, rubric‑threshold invariance, and ambiguity‑aware calibration—and applies them in a stress‑test protocol across four agent‑class judges using trajectories from ASSEBench and R‑Judge. The study finds that current judges often change verdicts in response to both meaningful normative shifts and meaningless structural rewrites, with up to 9.1% of verdicts flipping due to content‑preserving rewrites and 18‑43% of flips occurring on unambiguous cases, indicating that safety scores conflate agent actions with evaluator wording. whyItMatters":"The findings reveal a hidden failure mode in safety evaluation that could mislead researchers about agent performance, underscoring the need for policy‑invariant judges and transparent reporting protocols."

By Shihao Weng, Yang Feng, Xiaofei Xie
arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv Machine Learning
Sep 14

A False Average: Pooled CoT-Monitor Accuracy Conceals a Reasoning-Dependent Fragility

The paper demonstrates that aggregate accuracy figures for chain‑of‑thought (CoT) monitors can be misleading because a large portion of detected hacks rely solely on action patterns rather than reasoning. By rewriting only the agent’s reasoning to appear truthful while keeping actions identical, the authors show that the monitor’s performance on the reasoning‑dependent subset collapses dramatically, yet the overall pooled accuracy drops only modestly. The study reveals that CoT monitors are fragile when reasoning is the key signal and that accuracy should be reported separately for this subset.

By Shikhar Shiromani, Leo Richter
arXiv AI
Aug 26

Beyond Accuracy: A Dual-Judge Evaluation Protocol for Vision-Language Models in Legally Grounded Tasks

The paper introduces a dual‑judge evaluation protocol for vision‑language models in legally grounded tasks, pairing a 0‑10 quality judge with a strict binary semantic‑equivalence judge. Using a controlled UK traffic‑sign interpretation task, the authors analyze 4,680 evaluations across visibility and occlusion conditions, finding moderate association between judges and an asymmetric Type II error pattern that is most pronounced under heavy occlusion. The protocol requires only one additional LLM call and reveals quality‑trustworthiness signals that single‑judge methods miss.

By Su Myat Noe, Ha Thanh Nguyen, May Myo Zin, Ken Satoh