arXiv AI September 1, 2026 By Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment

Read the original on arXiv AI →

The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 4

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.

By Saket Reddy, Ke Yang, ChengXiang Zhai

llms reinforcement-learning fine-tuning benchmarks safety

Hugging Face Trending Papers

Jun 3

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates.

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 2

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.

By Hyung Gyu Rho

llms nlp reinforcement-learning

arXiv AI

1d ago

The Reward Model Selection Crisis in Personalized Alignment

arXiv:2512.23067v3 Announce Type: replace Abstract: Personalized alignment from preference data has focused primarily on improving personal reward model (RM) accuracy, with the implicit assumption th...

By Fady Rezk, Yuangang Pan, Chuan-Sheng Foo, Xun Xu, Nancy Chen, Henry Gouk, Timothy Hospedales

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Sep 30

Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving

arXiv:2609.36750v1 Announce Type: cross Abstract: Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods...

By Yiming Wang, Yikang Liu, Qingyuan Tian, Xingyu Chen, Zhuosheng Zhang, Zhaopeng Tu, Rui Wang

llms reinforcement-learning benchmarks

arXiv AI

3d ago

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

The paper introduces Collaborative Reward Modeling (CRM), an online framework that uses cross-model peer review and curriculum learning to identify reliable preference pairs for training reward models (RMs). By having two RMs evaluate each other's sample selections and synchronizing their learning progress, CRM reduces the impact of noisy feedback, improving RM generalization by up to 9.94 points on RewardBench under 40% preference noise. The approach also extends to implicit-reward alignment, demonstrating robustness and broad applicability for learning from imperfect feedback.

By Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang

llms reinforcement-learning safety

One email a morning, machine-written

Related stories

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

The Reward Model Selection Crisis in Personalized Alignment

Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment