Poolwise is a protocol that evaluates reward models for selecting training data across different prompts by choosing a fixed fraction of responses based on score and measuring the quality of the selected set with external labels. It reports within‑prompt ranking separately and shows that training each prompt’s mean score to match its average response rating improves selection over permuted or omitted mean targets in all tested seeds at 30% retention. In grouped HelpSteer experiments, the model with the highest within‑prompt accuracy selects a lower‑rated set than the correct‑mean model, and subtracting prompt means from reference responses preserves local rankings but reduces selected helpfulness for several reward models and judges.
whyItMatters":"Poolwise offers a direct method to compare reward models and score transformations for data selection, addressing the limitation of existing benchmarks that only compare responses within the same prompt."
By Mateusz Modrzejewski, Tomasz Radzikowski
The paper introduces Collaborative Reward Modeling (CRM), an online framework that uses cross-model peer review and curriculum learning to identify reliable preference pairs for training reward models (RMs). By having two RMs evaluate each other's sample selections and synchronizing their learning progress, CRM reduces the impact of noisy feedback, improving RM generalization by up to 9.94 points on RewardBench under 40% preference noise. The approach also extends to implicit-reward alignment, demonstrating robustness and broad applicability for learning from imperfect feedback.
By Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv:2610.05370v2 Announce Type: replace
Abstract: Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside...
By Xuancheng Li, Beining Wang, Haitao Li, Heng Wang, Yujia Zhou, Qingyi Pan, Blaze Chen, Yiqun Liu, Min Zhang, Qingyao Ai
The paper investigates why prompt optimization works better for some tasks than others by decomposing reward variance into response variance and system‑prompt variance. It finds that optimization succeeds when system‑prompt variance dominates, and that adding more user prompts can actually reduce this variance, especially on heterogeneous datasets. To address this, the authors propose $p1$, a filtering method that selects a small set of high‑variance user prompts, which improves optimization on reasoning benchmarks and even allows a system prompt trained on just two AIME 24 prompts to generalize well.
By Zhaolin Gao (Sid), Yu (Sid), Wang, Bo Liu, Thorsten Joachims, Kiant\'e Brantley, Wen Sun
arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.
By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
PerSPL (Personalized Semi-supervised Preference Learning) is a backbone‑agnostic framework that stabilizes cross‑prompt score coordinates using pair‑mean score centering, enabling the conversion of unlabeled responses into user‑specific training signals. It addresses the scarcity of user‑specific preference comparisons by leveraging large pools of unlabeled responses, which are otherwise unused by existing personalized reward models. Across four benchmarks—PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena—PerSPL achieves the highest mean accuracy for both seen and unseen users, demonstrating its effectiveness in summarization and open‑domain chatbot interaction settings.
By Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi