arXiv Machine Learning October 9, 2026 By Erik Skalnes, Layne C. Price, Raviteja Anantha, Michael Oberst

Evaluating Rubric Generation with Interventional Transfer

Read the original on arXiv Machine Learning →

The paper introduces Interventional Transfer (IT), a method for evaluating rubric generation by assessing how two rubrics respond to perturbations that cause a pass/fail shift. IT reveals asymmetries in rubrics generated by different LLMs when applied to HealthBench, showing that improvements on one rubric do not reliably transfer to the other. The study contrasts IT with existing evaluation approaches, highlighting its ability to uncover these asymmetries.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

Sep 3

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

The paper introduces RuVerBench, a benchmark with 2,458 instances for evaluating the reliability of Large Language Models acting as judges (LaaJ) in verifying rubric compliance within agentic scenarios such as deep research and agentic coding. It systematically meta‑evaluates frontier LLMs, revealing that even the most advanced models perform well yet still produce substantial noise. The study also examines how prompt design, batching, and majority voting affect verification accuracy, noting that weaker models are more prompt‑sensitive, batched verification trades accuracy for efficiency, and majority voting offers diminishing returns.

By Yangda Peng, Yunjia Qi, Haotian Xia, Guanzhong He, Xintong Shi, Richeng Xuan, Songyuanyi Lu, Yixian Liu, Zhichao Hu, Yuhong Liu, Hao Peng

llms agents benchmarks

arXiv AI

Sep 2

CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

arXiv:2609.00892v1 Announce Type: new Abstract: Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforceme...

By Siyuan Li, Xinxin Song, Chen Ruinian, Jingjing Fan, Tingxiong Xiao, Yangen Hu, Ke Zeng, Jinli Suo

llms reinforcement-learning benchmarks

arXiv Computation and Language

Sep 1

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

arXiv:2608.29856v1 Announce Type: new Abstract: Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during...

By Yifan Chen, Haitao Li, Qingyao Ai, Fengbin Zhu, Tat-Seng Chua, Min Zhang, Yiqun Liu

llms reinforcement-learning benchmarks

arXiv AI

Jul 23

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

arXiv:2604. 06996v2 Announce Type: replace-cross Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs.

By Jos\'e Pombal, Ricardo Rei, Andr\'e F. T. Martins

llms benchmarks safety

arXiv AI

Aug 13

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

arXiv:2608. 11669v1 Announce Type: cross Abstract: Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.

By Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu

llms reinforcement-learning benchmarks safety

arXiv AI

Aug 5

Rubrics as Privileged Information for Open-Ended Generation

arXiv:2608. 02948v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations.

By Deepika Bablani, Ajay Gupta, Wanming Chen

llms reinforcement-learning efficiency benchmarks

One email a morning, machine-written

Related stories

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Rubrics as Privileged Information for Open-Ended Generation