The study investigates how subtle changes in the presentation of a structured judging request can influence the behavior of an answer‑grading model. By adding a single colon to a candidate answer, the researchers found that the model’s false‑acceptance rate rose dramatically—from 1–3% to over 26%—across two benchmark datasets (DROP and GSM8K) and different grading configurations. The effect varied with the presentation format, showing that logically equivalent requests can lead to markedly different vulnerabilities.
arXiv:2609.26550v3 Announce Type: replace Abstract: LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge tha...
arXiv:2609.37616v1 Announce Type: new Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on th...
The paper introduces a two‑dimensional construct validity framework for evaluating large language models (LLMs) as judges, defining invariance (S) and sensitivity (R) to construct‑preserving and construct‑changing edits. Experiments across seven judges and four domains reveal high invariance (average S = 0.945) but low sensitivity (average R = 0.319), with sensitivity varying by edit type. Audits of public label sets show that surface‑only predictors can reproduce a substantial portion of labels, underscoring that high agreement does not guarantee construct validity.
The paper introduces a paired candidate‑coverage benchmark protocol for typed decision models, evaluating two models—Laya and Jev—on datasets such as AG News, DBpedia, Emotion, and TREC. It reports that Laya detects a high percentage of missing-answer cases but also falsely rejects many valid candidates, whereas Jev shows lower false rejection rates but also lower detection of missing answers. The study highlights the need for separate measurements of classification, score ranking, and rejection policies, noting that the benchmark is descriptive and limited to reference‑label omission.
The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.
arXiv:2609.06058v1 Announce Type: cross Abstract: As vision-language models (VLMs) become increasingly capable and are deployed in consequential real-world settings, they must evaluate evidence indep...
arXiv:2609.36587v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using...
The study investigates how deictic ambiguity—specifically the shifting reference of expressions like "previous"—affects Draft‑Verify‑Revise pipelines that use multiple large language models (LLMs). Using a synthetic dataset of 10 base examples and 21 reasoning‑effort configurations, six LLMs were evaluated for their ability to correctly resolve the ambiguous expression across the draft, verify, and revise stages. Results show wide variance in balanced accuracy, with GPT‑5.2 improving from 0.156 to 0.942 with increased reasoning effort, while Gemini 3 Pro consistently achieved high accuracy above 0.94 even at low reasoning effort, and meta‑evaluators often relied on surface cues when making errors.
The paper investigates whether activation interventions that guide a language model toward a target answer also enable the model to use the intended condition in other computations. Using Qwen3.5‑27B on tool‑use tasks, the authors find that residual‑writing variants improve target‑selection accuracy but these gains vanish when familiar facts and identifiers are recombined, and they do not translate into better independent judgments about eligibility or selection. The study demonstrates a clear separation between target‑selection control, condition‑sensitive judgments, and complete‑action correctness, highlighting the importance of evaluating how edited states influence downstream reasoning rather than just the final answer.
The paper argues that verbalized confidence—once viewed as overconfident and coarse—has become the preferred soft‑scoring method for LLM‑as‑a‑Judge on top‑tier proprietary models released after 2025. Experiments on SummEval, AggreFact, and HelpSteer2 across up to 18 LLMs show that log‑probabilities are no longer the best signal, and that adding an overconfidence advisory and self‑debate further improves calibration and robustness. The authors note that these enhancements incur little accuracy loss on post‑2025 models but do affect pre‑2025 ones, highlighting a compatibility shift in how confidence should be measured.
arXiv:2610.05094v1 Announce Type: cross Abstract: Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design...
The paper argues that relying solely on overall accuracy or F1 scores to validate automatic scorers in LLM evaluation can conceal prompt‑specific errors that distort comparative results. By auditing six models on 180 invented‑entity questions across six prompts, the authors show that a scorer’s overall F1 of .916 masks substantial variation—up to 33.3%—in error rates depending on the prompt. In a second audit, differences in measured prompt effects dropped dramatically when multiple raters were used, highlighting the need to validate the exact comparison rather than the scorer alone.