The paper investigates how reward models, used for preference optimization and ranking, can be biased by prompt-specific effects that are not captured by pairwise evaluation. By comparing between‑prompt variance in model scores to human ratings, the authors find that reward scores cluster strongly by prompt, often more so than human judgments. They propose a human‑referenced audit that examines between‑prompt variation, sensitivity to incorrect prompts, and usefulness for response selection, and demonstrate that adjusting for prompt means can reduce misleading global selection of responses.
By Mateusz Modrzejewski, Tomasz Radzikowski
arXiv:2609.36652v1 Announce Type: new
Abstract: Open-ended generation lacks canonical answers, making pointwise rewards difficult to calibrate for group-based reinforcement learning. Directly ranking...
By Zixuan Yang, Yiqun Chen, Qi Liu, Wei Yang, Erhan Zhang, Liyi Chen, Qimeng Wang, Yan Gao, Jiaxin Mao
arXiv:2608.30005v1 Announce Type: new
Abstract: Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific c...
By Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao
arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.
By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
arXiv:2610.05370v2 Announce Type: replace
Abstract: Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside...
By Xuancheng Li, Beining Wang, Haitao Li, Heng Wang, Yujia Zhou, Qingyi Pan, Blaze Chen, Yiqun Liu, Min Zhang, Qingyao Ai
PerSPL (Personalized Semi-supervised Preference Learning) is a backbone‑agnostic framework that stabilizes cross‑prompt score coordinates using pair‑mean score centering, enabling the conversion of unlabeled responses into user‑specific training signals. It addresses the scarcity of user‑specific preference comparisons by leveraging large pools of unlabeled responses, which are otherwise unused by existing personalized reward models. Across four benchmarks—PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena—PerSPL achieves the highest mean accuracy for both seen and unseen users, demonstrating its effectiveness in summarization and open‑domain chatbot interaction settings.
By Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi
The paper introduces Collaborative Reward Modeling (CRM), an online framework that uses cross-model peer review and curriculum learning to identify reliable preference pairs for training reward models (RMs). By having two RMs evaluate each other's sample selections and synchronizing their learning progress, CRM reduces the impact of noisy feedback, improving RM generalization by up to 9.94 points on RewardBench under 40% preference noise. The approach also extends to implicit-reward alignment, demonstrating robustness and broad applicability for learning from imperfect feedback.
By Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv:2409.00696v4 Announce Type: replace-cross
Abstract: Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs)....
By Jasper Dekoninck, Maximilian Baader, Martin Vechev
Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of re...
The paper introduces a novel framework that combines vision‑language model (VLM) generated preferences with the Plackett‑Luce (PL) ranking model for reward learning in reinforcement learning. Unlike traditional pairwise Bradley‑Terry approaches, the PL formulation allows listwise rankings of multiple candidates, enabling the use of different ranking sizes (K = 3, 4, 5). Experiments on Meta‑World manipulation tasks show that PL‑based reward models train robotic policies as effectively as, or better than, pairwise, K‑wise, and RL‑VLM‑F baselines, achieving up to an 86% mean final success rate and matching the Oracle baseline on the Drawer Open task.
By Srivalli Katkuri, Maxwell Kawada, Juan Wachs
arXiv:2605.26958v2 Announce Type: replace-cross
Abstract: Reinforcement learning in open-ended long-form generation is challenging because reliable reference answers and automatic metrics are often u...
By Zixuan Yang, Yiqun Chen, Wei Yang, Erhan Zhang, Zihan Shen, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Jiaxin Mao
arXiv:2606. 03980v1 Announce Type: new Abstract: Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines.
By Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang