arXiv Computation and Language By Rohith Reddy Bellibatlu, Edward Raff, Wenbin Zhang

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

Read the original on arXiv Computation and Language →

JudgeSense is a benchmark comprising 880 items from human‑labelled corpora, each presented under two differently worded instructions that ask the same question. The study evaluates 25 judges from six providers across four tasks, measuring how rewording affects agreement with the judge’s own verdicts. Results show that rewording reduces agreement on all tasks, with significant effects on two, and that stability varies across tasks and is not predicted by parameter count.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Nov 5

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

The paper evaluates 21 LLM-as-a-Judge models from nine providers across MT-Bench, JudgeBench, and RewardBench, using agreement, consistency, and bias audit protocols over 118 runs and about 541,000 judgments. It finds that exact-match agreement overstates performance, with Cohen’s kappa lower by 33–41 percentage points, judge rankings shift up to 14 positions, and high test–retest reliability coexists with significant position bias in some production judges. Verbosity bias is minimal across the cohort. The authors propose a Minimum Viable Validation Protocol based on these findings.

By Justin D. Norman, Michael U. Rivera, D. Alex Hughes
arXiv Machine Learning
Oct 23

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

The study investigates how different prompting strategies affect large language model (LLM) judges across multiple languages and tasks. Using a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks, the authors collected over 1 million pointwise scores validated against human judgments. Results show that prompting effectiveness is highly dependent on the judge model, can reverse direction across judges, and becomes less stable in low‑resource languages, indicating that prompt choice must be tailored to the specific evaluation setting.

By Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro
arXiv AI
Oct 2

The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating

The paper demonstrates that reading a large language model (LLM) judge’s verdict from the logits of its first generated token—an approach used in constrained decoding and likelihood‑scoring evaluation—introduces a significant distortion in position bias. Because judges do not always start with a verdict token (12–49% of cases for Qwen3 judges and <3% for Llama‑3.1‑8B and Phi‑3.5‑mini), this readout often returns the first response rather than a true judgment, inflating position bias by up to 42 points while barely affecting judge accuracy. The authors recommend reporting the frequency with which a judge leads with a verdict token to provide a more accurate assessment of position bias.

By Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
arXiv AI
Sep 28

When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess

The paper investigates when a language‑model judge can truly ground its verdicts in code correctness. It shows that current multi‑agent verification methods rely on evidence that is both independent of the answer and distinct between candidates—conditions that fail in code judging. By analyzing two label‑free measurements from the judge’s logs, the authors demonstrate that gating on one measurement allows the system to decline uncertain comparisons, improving accuracy from 20.7% to 36.9% while still answering half of all cases.

By Salma Roshdy Aly, Hussein Assaf, Ziad Kobti