← Back to all news
arXiv AI By Su Myat Noe, Ha Thanh Nguyen, May Myo Zin, Ken Satoh

Beyond Accuracy: A Dual-Judge Evaluation Protocol for Vision-Language Models in Legally Grounded Tasks

Read the original on arXiv AI →

The paper introduces a dual‑judge evaluation protocol for vision‑language models in legally grounded tasks, pairing a 0‑10 quality judge with a strict binary semantic‑equivalence judge. Using a controlled UK traffic‑sign interpretation task, the authors analyze 4,680 evaluations across visibility and occlusion conditions, finding moderate association between judges and an asymmetric Type II error pattern that is most pronounced under heavy occlusion. The protocol requires only one additional LLM call and reveals quality‑trustworthiness signals that single‑judge methods miss.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
2d ago

SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents

SAGE (State‑Grounded, Abstention‑Aware Evaluation) is a new framework for assessing task‑oriented dialogue agents that focuses on whether each turn correctly advances the workflow state rather than just the quality of the reply. It compiles workflow specifications and per‑turn state differences into schema‑grounded criteria, then evaluates them through a cascade of symbolic rules and encoder/NLI verifiers that abstain instead of guessing, producing a turn‑level decision with an evidence trace. In experiments across MultiWOZ, Schema‑Guided Dialogue, and ABCD datasets, SAGE‑Core—using only symbolic rules and on‑device encoders—outperforms all evaluated LLM‑based judges, including GPT‑4.1 variants, while incurring zero paid LLM cost.

By Rayan Khoury, Shih-Yao Lin, Pratyush Mishra
llmsagents
More like this →