EvoRubric introduces a co‑evolutionary reinforcement learning framework that learns rubric‑driven evaluation criteria for open‑ended text generation. The system uses a shared policy that simultaneously generates responses and rubrics, while a frozen meta‑verifier and grader provide validity checks and scoring. Experiments on medical, writing, and science benchmarks show that EvoRubric outperforms static and other evolving‑rubric baselines, achieving higher average scores on five tasks.
By Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, Jiuxin Cao
The paper introduces Interventional Transfer (IT), a method for evaluating rubric generation by assessing how two rubrics respond to perturbations that cause a pass/fail shift. IT reveals asymmetries in rubrics generated by different LLMs when applied to HealthBench, showing that improvements on one rubric do not reliably transfer to the other. The study contrasts IT with existing evaluation approaches, highlighting its ability to uncover these asymmetries.
By Erik Skalnes, Layne C. Price, Raviteja Anantha, Michael Oberst
arXiv:2608.29856v1 Announce Type: new
Abstract: Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during...
By Yifan Chen, Haitao Li, Qingyao Ai, Fengbin Zhu, Tat-Seng Chua, Min Zhang, Yiqun Liu
arXiv:2608. 02948v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations.
By Deepika Bablani, Ajay Gupta, Wanming Chen
Writer‑R1 is a writing training pipeline that uses Inductive Rubric Generation (IRG) to automatically construct prompt‑specific rubrics for open‑ended writing tasks. The pipeline expands an open writing corpus, applies the generated rubrics during supervised training and reinforcement learning, and provides the rubrics to writers before generation. A compact model, RubricGen‑4B, can supply rubrics for new requests, and Writer‑R1 improves WritingBench and HelloBench scores over 4B and 8B backbones.
By Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji, Chunlai Zhou, Biao Qin
The paper introduces a retrieval‑augmented multi‑agent framework that automatically generates instance‑specific evaluation rubrics for medical language models. By retrieving authoritative medical evidence, decomposing it into atomic facts, and combining these with user interaction constraints, the system produces fine‑grained criteria that outperform GPT‑4o on HealthBench and LLMEval‑Med. The generated rubrics also guide response refinement, improving medical LLM output quality by 9.2%.
By Yinzhu Chen, Abdine Maiga, Hossein A. Rahmani, Emine Yilmaz
arXiv:2606. 09165v1 Announce Type: new Abstract: Safety judges are increasingly deployed to evaluate model outputs against evolving criteria, yet recent meta-evaluation work shows they remain brittle under prompt and rubric variation, with false negative-rate swings of up to 0.
By Yongtaek Lim, Hyeji Choi, Minwoo Kim
arXiv:2608. 14684v1 Announce Type: cross Abstract: LLM judges increasingly evaluate responses against fine-grained rubric checklists.
By Dingyao Yu, Tong Zhang, Yutao Mou, Yunxiao Zhang, Wei Ye, Shikun Zhang
arXiv:2606. 05174v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown strong promise in healthcare applications.
By Arash Ahmadi, Parisa Masnadi, Sarah Sharif, Charles Nicholson, David Ebert, Mike Banad
arXiv:2606. 09118v1 Announce Type: new Abstract: As LLM capabilities advance rapidly, the evaluation methods used to assess them increasingly lag behind.
By Sushant Mehta, Liudas Panavas, Edwin Chen
arXiv:2608.30005v1 Announce Type: new
Abstract: Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific c...
By Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao
The paper introduces ReVeR, a pointwise rubric reward model that derives hard rules and graded principles directly from instructions, avoiding response-induced bias. It employs tool-augmented judges and label‑blind distillation to create ReVeR‑Traces, a large dataset of rubric reasoning and judging trajectories. Experiments show ReVeR surpasses pairwise rubric models in preference accuracy, especially for instruction following, and reveal that higher preference accuracy does not always lead to better downstream policies.