Rubric-based evaluation of large language models (LLMs) often assigns binary judgments to multiple criteria, but the certainty of these judgments varies widely. Existing aggregation methods treat all criteria as equally certain, potentially misleading reinforcement learning signals. The authors propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that uses conformal calibration to quantify and weight criterion-level uncertainty, improving preference accuracy and downstream RL alignment.
By Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen, Yichen Sun, Yuan Lu, Xiaoxi Li, Hao Wang, Zhixuan Chu, Kui Ren
The paper introduces ReVeR, a pointwise rubric reward model that derives hard rules and graded principles directly from instructions, avoiding response-induced bias. It employs tool-augmented judges and label‑blind distillation to create ReVeR‑Traces, a large dataset of rubric reasoning and judging trajectories. Experiments show ReVeR surpasses pairwise rubric models in preference accuracy, especially for instruction following, and reveal that higher preference accuracy does not always lead to better downstream policies.
arXiv:2610.05370v2 Announce Type: replace
Abstract: Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside...
By Xuancheng Li, Beining Wang, Haitao Li, Heng Wang, Yujia Zhou, Qingyi Pan, Blaze Chen, Yiqun Liu, Min Zhang, Qingyao Ai
The paper introduces JudgeBiasBench, a benchmark that systematically quantifies judgment biases in large language model (LLM)-based judges across four dimensions and 12 bias types. It evaluates both generative and discriminative judges, revealing significant bias patterns that undermine reliability. The authors propose bias-aware training—reinforcement learning for generative judges and contrastive learning for discriminative judges—to reduce these biases while maintaining evaluation performance.
By Hongli Zhou, Hui Huang, Rui Zhang, Kehai Chen, Bing Xu, Conghui Zhu, Tiejun Zhao, Muyun Yang
arXiv:2609.23457v1 Announce Type: new
Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, towa...
By Hao Li, Zhengkun Zhang, Gangqiang Hu, Zhen Zhang, Yude Gao, Dai Dai, Jing Liu
arXiv:2603. 05659v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers.
By Wisdom Ikezogwo, Mehmet Saygin Seyfioglu, Ranjay Krishna, Karim Bouyarmane
Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult. Existing approache...
arXiv:2607. 01830v1 Announce Type: new Abstract: Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks.
By Dazhi Fu, Jiuding Yang, Yiwen Guo, Jicong Fan
arXiv:2606. 03980v1 Announce Type: new Abstract: Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines.
By Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang
arXiv:2608. 02181v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets.
By Zhijian Zhou, Long Li, Xuan Zhang, Zongkai Liu, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, Yuan Qi
Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator.
arXiv:2608. 16156v1 Announce Type: new Abstract: Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult.
By Huan Zhang, Mingju Chen, Dongxu Zhou, Can Lv, Heng Chang, Sen Cui, Faguo Wu, Shiji Zhou