arXiv:2610.14707v2 Announce Type: new
Abstract: Expert-aligned "taste" benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad bench...
By Xiaofan Wu, Xi Zeng, Miaoxia Chen, Peishan Chen, Shuyan Li, Chen Wang, Xiaohan Wang, Wanling Xu, Jiyun Yao, Hanyong Zhong, Jiahao Zhu
The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.
By Idil Gozel
The study investigates whether Jev, a typed classifier that outputs probabilities over allowed answers without generating text, can replace large language model (LLM) rubric judges. Across nine panels from seven benchmarks, Jev’s accuracy differed significantly from LLM judges in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons were inconclusive. In terms of cost and speed, Jev was 29 to 325 times cheaper and 30 to 220 times faster than the flash‑tier LLM judges, and a cascade approach that defers uncertain Jev verdicts to an LLM yielded only modest gains.
whyItMatters":"The findings suggest that a lightweight classifier like Jev can serve as an efficient first‑stage evaluator, potentially reducing the reliance on expensive and slow LLM judges in automated grading pipelines."
By Delip Rao, Chris Callison-Burch
arXiv:2610.07177v1 Announce Type: cross
Abstract: An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of...
By Gowthamkumar Nandakishore
LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that...
arXiv:2607. 02104v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.
By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao
The study compares Jev, a typed classifier that outputs probabilities over allowed answers, with three flash-tier LLM rubric judges across nine panels from seven benchmarks. Jev’s accuracy differs significantly from an LLM judge in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons are inconclusive. In terms of cost and speed, Jev is 29 to 325 times cheaper and 30 to 220 times faster than the LLM judges, and a cascade that defers uncertain Jev verdicts to an LLM yields only modest gains of up to 2.0 points over the best single judge.
The article examines how classical test theory reliability statistics misrepresent the performance of large language model (LLM) judges. It shows that internal‑consistency coefficients, the dependability index, and Livingston‑Lewis accuracy each conflate judge error with item design or criterion validity, making it impossible to attribute a single reliability value to the judge alone. The authors argue that such misattribution can influence deployment decisions and documentation.
By Louis Yiven Zhu
The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.
By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov
The paper investigates how the scoring rules used in multimodal large‑language‑model (LLM) benchmarks can dramatically alter reported accuracy. By reproducing official scorers for eight benchmarks and comparing them to a blinded human‑plus‑LLM adjudication, the authors show that the same model outputs can yield accuracy differences of up to 22 points depending on the rule. They argue that no single scoring rule is reliable and recommend that benchmarks report non‑compliance rates and use at least two distinct scoring rules.
By Ahmad Jarjis Hasan, Jonathan Lwowski
The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.
By Alden Do Rosario, Kirollos Atef
The paper introduces Conditional Accuracy Profiling (CAP), a diagnostic framework that breaks down LLM‑judge accuracy into eight deployment conditions covering content sensitivity, robustness, and rationale quality. CAP is benchmark‑agnostic and can be applied directly when annotations are available, via task‑subset proxies, or through controlled augmentation. Applied to seven judges across six benchmarks, CAP reveals that aggregate accuracy masks important differences—for example, judges highly sensitive to omitted qualifications rank poorly on a Hard‑Constructed subset, and Position Robustness is generally stable but can degrade sharply under adversarial stress.
By Wenqi Li, Bin Liu, Mindi Ruan, Chuanbo Hu, Minglei Yin, Xin Li