← Back to all news
arXiv Machine Learning By Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi

PerSPL: Personalized Semi-supervised Preference Learning

Read the original on arXiv Machine Learning →

PerSPL (Personalized Semi-supervised Preference Learning) is a backbone‑agnostic framework that stabilizes cross‑prompt score coordinates using pair‑mean score centering, enabling the conversion of unlabeled responses into user‑specific training signals. It addresses the scarcity of user‑specific preference comparisons by leveraging large pools of unlabeled responses, which are otherwise unused by existing personalized reward models. Across four benchmarks—PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena—PerSPL achieves the highest mean accuracy for both seen and unseen users, demonstrating its effectiveness in summarization and open‑domain chatbot interaction settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 27

Learning to summarize user information for personalized reinforcement learning from human feedback

The paper introduces PLUS, a framework that uses reinforcement learning to generate text-based summaries of individual users’ preferences, characteristics, and past conversations. These summaries condition a reward model, allowing it to predict personalized response preferences and improving reward accuracy by 11–77 % over the standard Bradley‑Terry model. PLUS demonstrates robust performance with new users and topics, achieves a 25 % improvement over existing personalized RLHF techniques, and enables zero‑shot personalization for state‑of‑the‑art models like GPT‑4.

By Hyunji Nam, Yanming Wan, Mickel Liu, Peter Ahnn, Jianxun Lian, Natasha Jaques
llmsnlpreinforcement-learningbenchmarkssafety
More like this →
arXiv Machine Learning
Jul 31

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.

By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv AI
3d ago

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

The paper introduces Collaborative Reward Modeling (CRM), an online framework that uses cross-model peer review and curriculum learning to identify reliable preference pairs for training reward models (RMs). By having two RMs evaluate each other's sample selections and synchronizing their learning progress, CRM reduces the impact of noisy feedback, improving RM generalization by up to 9.94 points on RewardBench under 40% preference noise. The approach also extends to implicit-reward alignment, demonstrating robustness and broad applicability for learning from imperfect feedback.

By Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang
llmsreinforcement-learningsafety
More like this →