arXiv Machine Learning By Louis Yiven Zhu

Three Ways Classical Test Theory Misleads for LLM Judges

Read the original on arXiv Machine Learning →

The article examines how classical test theory reliability statistics misrepresent the performance of large language model (LLM) judges. It shows that internal‑consistency coefficients, the dependability index, and Livingston‑Lewis accuracy each conflate judge error with item design or criterion validity, making it impossible to attribute a single reliability value to the judge alone. The authors argue that such misattribution can influence deployment decisions and documentation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language
Oct 1

Three Ways Classical Test Theory Can Mislead About LLM Judges

The article examines how classical test theory statistics—Kuder‑Richardson coefficient, dependability index, and Livingston‑Lewis accuracy—can mislead when applied to large language model (LLM) judges that are evaluated with a single prompt and no gold labels. Using Claude Haiku 4.5 on 210 short‑answer items, the authors show that these metrics fail to isolate the judge’s performance because the judge’s single administration provides no variance component. They argue that reliable statements about an LLM judge require gold labels or varied scorer facets, and that bank design heavily influences reliability estimates.

By Louis Yiven Zhu
arXiv AI
Sep 24

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

The study analyzes 373,019 judgments from LLM‑scored benchmarks, decomposing variance into system, item, judge, and interaction components via generalizability theory. It finds that with a single judge, generalizability converges to a ceiling determined by the system‑by‑judge variance, which is substantially lower in pairwise preference settings, allowing one judge to suffice. The research also reveals significant biases in presentation order and highlights that many published win‑rate claims fall below the measured floor of the benchmarks.

By Atul Anand
arXiv AI
Nov 6

Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test

The paper investigates whether large language model (LLM) safety judges are truly policy‑invariant, meaning their verdicts should depend only on the agent’s behavior and not on how the evaluation policy is phrased. It introduces three testable principles—rubric‑semantics invariance, rubric‑threshold invariance, and ambiguity‑aware calibration—and applies them in a stress‑test protocol across four agent‑class judges using trajectories from ASSEBench and R‑Judge. The study finds that current judges often change verdicts in response to both meaningful normative shifts and meaningless structural rewrites, with up to 9.1% of verdicts flipping due to content‑preserving rewrites and 18‑43% of flips occurring on unambiguous cases, indicating that safety scores conflate agent actions with evaluator wording. whyItMatters":"The findings reveal a hidden failure mode in safety evaluation that could mislead researchers about agent performance, underscoring the need for policy‑invariant judges and transparent reporting protocols."

By Shihao Weng, Yang Feng, Xiaofei Xie
arXiv AI
1d ago

Screen Before You Interpret: A Portable Validity Screen for Benchmark-Based LLM Confidence Signals

The paper introduces a portable validity screen for evaluating large language model (LLM) confidence signals, mirroring clinical personality assessment practices. It computes metrics such as the rate of high‑confidence errors (L), low‑confidence correct answers (Fp), and their contrast (RBS), without adding new statistics, to determine whether a signal truly tracks correctness. Applied to 20 frontier models on 452 two‑alternative items and 33 models on 1,500 MMLU items, the screen classifies models as Valid, Indeterminate, or Invalid, revealing that many models fail to meet validity criteria depending on confidence thresholds.

By Jon-Paul Cacioli