Rubric Curriculum RL (RcRL) is a reinforcement learning approach that leverages the generation‑verification gap in non‑verifiable domains by using pairwise preferences over rollouts and a curriculum over rubric criteria. Unlike traditional methods that rely on absolute scoring and stationary judges, RcRL provides a more discriminative, non‑stationary signal, allowing policies to maintain output entropy and continue improving beyond 1,000 training steps. In experiments on creative writing and HealthBench, RcRL outperforms baseline models, achieving a 70.5% human win rate and significant accuracy gains over the base model.
By Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
The paper introduces Rubric Dropout, a technique that randomly removes a subset of rubric criteria during training to prevent reinforcement learning agents from exploiting the rubric as a proxy for quality. By resampling which criteria are rewarded at each step and ensuring all responses to a prompt are evaluated on the same subset, the method reduces reward hacking and improves out‑of‑distribution performance on medical and science benchmarks. Block Dropout further groups semantically similar criteria to avoid near‑duplicate rewards, yielding additional gains.
By Minglai Yang, Xinyu Guo, Zhengliang Shi, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Dan Roth, Ying Liu
The paper introduces Step-wise Rubrics as Rewards (SRaR), a reinforcement learning framework that assigns rubric-based scores to individual reasoning steps in large language models. By normalizing step scores and combining them with outcome rewards through a decoupled advantage estimator, SRaR preserves criterion-level structure and mitigates issues like rewarding flawed steps or encouraging endless self‑correction. Experiments on 16K problems and six math benchmarks show that SRaR outperforms prior rubric methods, improving accuracy, increasing faithful reasoning rates, and reducing self‑correction loops.
By Weichu Xie, Haozhe Zhao, Wenpu Liu, Yongfu Zhu, Liang Chen, Minghao Ye, Zirong Chen, Yuqi Xu, Shuai Dong, Ziyue Wang, Xinbo Xu, Kean Shi, Ruoyu Wu, Xiaoying Zhang, Wenqi Shao, Baobao Chang, Nan Duan, Jiaqi Wang
arXiv:2608.30005v1 Announce Type: new
Abstract: Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific c...
By Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao
arXiv:2610.19525v1 Announce Type: new
Abstract: Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underly...
By Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim
The paper introduces ReVeR, a pointwise rubric reward model that derives hard rules and graded principles directly from instructions, avoiding response-induced bias. It employs tool-augmented judges and label‑blind distillation to create ReVeR‑Traces, a large dataset of rubric reasoning and judging trajectories. Experiments show ReVeR surpasses pairwise rubric models in preference accuracy, especially for instruction following, and reveal that higher preference accuracy does not always lead to better downstream policies.
arXiv:2610.02824v1 Announce Type: new
Abstract: Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requireme...
By Yuxuan Fan, Jaehong Yoon
arXiv:2607. 04332v1 Announce Type: new Abstract: In this paper, we consider the setting where large language models (LLMs) are trained using reinforcement learning (RL) to simultaneously improve reasoning accuracy and verbalize its confidence.
By Chee Heng Tan, Zhuoyi Lin, Mehul Motani, Wee Sun Lee
The paper introduces a two‑stage training framework for rubric‑based reinforcement learning (RL) that first uses rubric‑privileged on‑policy distillation (RP‑OPD) to provide dense token‑level supervision, then applies RL to optimize the rubric reward. In RP‑OPD, a student model learns to match a rubric‑aware teacher’s next‑token distributions on student‑generated prefixes. Experiments on HealthBench, ResearchQA, and RubricHub Science show that this two‑stage approach yields the highest scores and reduces reward hacking compared to a standard SFT + RL baseline.
By Xinpeng Wang, Wei Shi, Yu-Chia Chen, Maria Zontak, Yun He, Richard Yuanzhe Pang
arXiv:2610.26932v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents in non-verifiable domains such as financial services, where they must int...
By Ashish Bansal, Hossam Amer, Keerat Kaur Guliani, Parinita Edke, Lesley Matheson, Leizer Teran
Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. W...
arXiv:2610.17733v1 Announce Type: cross
Abstract: Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large lan...
By Guangcheng Zhu, Yingfan Ma, Xing Zheng, Haobo Wang, Shenzhi Yang, Kai Tang, Zhengqing Zang, Bowen Song, Weiqiang Wang, Gang Chen