← Back to all news
arXiv Computation and Language By Mateusz Modrzejewski, Tomasz Radzikowski

Poolwise: Evaluating Reward Models for Selection across Prompts

Read the original on arXiv Computation and Language →

Poolwise is a protocol that evaluates reward models for selecting training data across different prompts by choosing a fixed fraction of responses based on score and measuring the quality of the selected set with external labels. It reports within‑prompt ranking separately and shows that training each prompt’s mean score to match its average response rating improves selection over permuted or omitted mean targets in all tested seeds at 30% retention. In grouped HelpSteer experiments, the model with the highest within‑prompt accuracy selects a lower‑rated set than the correct‑mean model, and subtracting prompt means from reference responses preserves local rankings but reduces selected helpfulness for several reward models and judges. whyItMatters":"Poolwise offers a direct method to compare reward models and score transformations for data selection, addressing the limitation of existing benchmarks that only compare responses within the same prompt."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Oct 23

What Pairwise Evaluation Misses: Reward Scores across Prompts

The paper investigates how reward models, used for preference optimization and ranking, can be biased by prompt-specific effects that are not captured by pairwise evaluation. By comparing between‑prompt variance in model scores to human ratings, the authors find that reward scores cluster strongly by prompt, often more so than human judgments. They propose a human‑referenced audit that examines between‑prompt variation, sensitivity to incorrect prompts, and usefulness for response selection, and demonstrate that adjusting for prompt means can reduce misleading global selection of responses.

By Mateusz Modrzejewski, Tomasz Radzikowski
More like this →
arXiv Machine Learning
Jul 31

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.

By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 22

PerSPL: Personalized Semi-supervised Preference Learning

PerSPL (Personalized Semi-supervised Preference Learning) is a backbone‑agnostic framework that stabilizes cross‑prompt score coordinates using pair‑mean score centering, enabling the conversion of unlabeled responses into user‑specific training signals. It addresses the scarcity of user‑specific preference comparisons by leveraging large pools of unlabeled responses, which are otherwise unused by existing personalized reward models. Across four benchmarks—PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena—PerSPL achieves the highest mean accuracy for both seen and unseen users, demonstrating its effectiveness in summarization and open‑domain chatbot interaction settings.

By Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi
nlpreinforcement-learningbenchmarks
More like this →