The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.
By Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani
Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates.
arXiv:2608. 00175v1 Announce Type: new Abstract: Deep reinforcement learning (RL) agents achieve strong performance by optimizing scalar reward functions.
By Umer Siddique, Peilang Li, Conor Wallace, Yongcan Cao
arXiv:2607. 03248v1 Announce Type: cross Abstract: The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization.
By Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li
The paper introduces MARBO, a framework that grounds large language model agents’ actions and speech in relational beliefs about hidden roles and team alignments in social deduction games. By providing preference feedback only when behaviors are supported by reliable relational beliefs and lead to favorable outcomes, MARBO encourages consistent learning under uncertainty. Experiments demonstrate that compact LLM agents using MARBO consistently outperform existing baselines in representative social deduction games.
By Hwang Yechan, Bae Sangjun, Kim Jeongmo, Bang Sangwoo, Han Seungyul
arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.
By Saket Reddy, Ke Yang, ChengXiang Zhai
arXiv:2607. 20515v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences.
By Jingyi Huang, Ruohan Zong, Yujun Feng, Liran Ma, Lanyu Shang, Yang Zhang
arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.
By Xiaohua Feng, Yuyuan Li, Huwei Ji, Jiaming Zhang, Li Zhang, Tianyu Du, Chaochao Chen
The paper proposes shifting AI agent training from isolated task completion to collaborative interaction, defining three key dimensions—Productivity, Proactivity, and Personalization (PPP). It introduces UserVille, an environment with LLM-based user simulators and user-centric feedback, and a multi-objective reinforcement learning framework that optimizes PPP using rewards from task outcomes, question effort, and preference adherence. Experiments on SWE-Bench and BrowseComp-Plus show PPP-trained agents outperform strong LLM baselines, ask more targeted questions, and generalize to unseen preferences and tasks, with a user study underscoring the value of user-centric feedback for effective, supervised collaboration.
By Weiwei Sun, Xuhui Zhou, Weihua Du, Xingyao Wang, Sean Welleck, Graham Neubig, Maarten Sap, Yiming Yang
The paper introduces CurriPO, a tree‑structured curriculum that adapts to diverse user reward models in AI alignment. By automatically building a curriculum that branches and reuses reward models, it addresses the problem of users whose reward models are hard to optimize, a group often underserved by conventional methods. Experiments on personalized continuous control demonstrate that CurriPO improves population satisfaction by 1.2–2.1× over the best baseline while cutting training time.
arXiv:2608. 09507v1 Announce Type: cross Abstract: Natural language user preferences provide an interpretable interface for LLM personalization.
By Yuting Liu, Wei Wu, Jianzhe Zhao, Guibing Guo
arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.
By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao