The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.
By Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani
arXiv:2605. 01642v2 Announce Type: replace Abstract: Prevailing alignment methods target a fixed set of preferences and therefore risk forcing value lock-in as societal norms evolve over time.
By Rachel Freedman
MiCRo is a two‑stage framework that improves personalized preference learning for large language models. It first uses a context‑aware mixture model to capture diverse human preferences from large binary preference datasets, then applies an online routing strategy to dynamically adjust mixture weights based on context, reducing ambiguity. Experiments on multiple datasets show that MiCRo captures diverse preferences and enhances downstream personalization.
By Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, Han Zhao
arXiv:2506. 13741v2 Announce Type: replace-cross Abstract: Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions.
By Brahim Driss, Alex Davey, Riad Akrour
arXiv:2509. 22851v4 Announce Type: replace-cross Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks.
By Yaswanth Chittepu, Prasann Singhal, Greg Durrett, Scott Niekum
arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.
By Lucio La Cava, Andrea Tagarelli
arXiv:2606. 04284v1 Announce Type: cross Abstract: Preference modeling plays a central role in reinforcement learning from human feedback (RLHF), enabling large language models (LLMs) to align with human values.
By Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg
arXiv:2508. 16947v2 Announce Type: replace-cross Abstract: Despite significant progress, imitation learning-based autonomous driving planners remain largely restricted to reproducing high-frequency biased behaviors, overlooking the inherent behavioral diversity of human driving.
By Fan Ding, Xuewen Luo, Fucai Ke, Hwa Hui Tew, Susilawati Susilawati, Vishnu Monn Baskaran, Junn Yong Loo
The paper introduces PARED, an approach that uses inverse reinforcement learning to derive an explicit, interpretable reward from expert demonstrations. By training a lightweight discriminator on response-level features, PARED can guide both inference-time reranking and on-policy reinforcement learning, improving alignment for models that have not undergone supervised fine-tuning and further refining those that have. The method also supports contextual alignment, enabling a single policy to adapt its responses to different audience preferences.
By Micha{\l} Wili\'nski, Liu Leqi, Chirag Nagpal
arXiv:2506. 13702v4 Announce Type: replace-cross Abstract: Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback.
By Bilal Faye, Hanane Azzag, Mustapha Lebbah
arXiv:2505. 10892v2 Announce Type: replace Abstract: Post-training LLMs with RLHF and preference optimization methods (e.
By Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
arXiv:2504.20106v4 Announce Type: replace
Abstract: Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessiv...
By Ren-Wei Liang, Chin-Ting Hsu, Chan-Hung Yu, Saransh Agrawal, Shih-Cheng Huang, Chieh-Yen Lin, Shang-Tse Chen, Kuan-Hao Huang, Shao-Hua Sun