← Back to all news
arXiv AI By Ye Chen, Weining Zhang

Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation

Read the original on arXiv AI →

The paper investigates two strategies for improving large language model (LLM) evaluation: specialized judge weights and rule‑based deferral policies. Experiments on nearly 100,000 rubric‑conditioned samples show that correct rubrics boost accuracy, while incorrect ones hurt it, and that splitting training data into criterion‑specific experts can severely degrade performance unless the experts are warm‑started from a unified model. The authors demonstrate that lightweight deferral cascades can match or exceed the accuracy of larger standalone judges at a fraction of the compute cost, and they provide practical design rules for building efficient, reliable LLM evaluators.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 13

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.

By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv Computation and Language
5d ago

Stopping and Routing LLM Judge Panels

arXiv:2608. 19802v1 Announce Type: new Abstract: LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers.

By Bin Zhu, Yi Xie, Yanghui Rao
llmsnlpsafety
More like this →
arXiv AI
2d ago

No Judgment Without a Reason: Counterfactual Receipts for Versioned AI Evaluators

The paper introduces a framework for evaluating AI systems that not only checks final labels but also tracks the reasoning behind them through three core sources—grounds, norms, and authority—forming an eight-cell counterfactual judgment cube. It defines minimal source replacement sets, called judgment receipts, to explain changes in verdicts and provides certification cost bounds for black-box evaluators. The authors present ReasonBench, a benchmark with 19,520 cases, and demonstrate that while high standard accuracy can mask robustness issues, receipt accuracy reveals significant gaps in reasoning consistency across different models.

By Ye Chen, Weining Zhang
agentsbenchmarks
More like this →