arXiv Computation and Language By Constantinos Karouzos, Nikolaos Aletras

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters

Read the original on arXiv Computation and Language →

LambdaMix is a parameter‑efficient method that learns a family of policies conditioned on user preferences by jointly training a low‑rank adapter for each objective over a frozen base model. A single preference vector selects an adapter mixture to generate responses and weight normalized rewards, enabling on‑line reinforcement learning without further optimization at inference. Experiments on four models show LambdaMix outperforms baselines in helpfulness, harmlessness, math problem solving, and comment rewriting, while using fewer training completions and storing fewer adapter parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Oct 5

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

The paper introduces GAW-PO, a gradient‑aligned token reweighting technique for Direct Preference Optimization (DPO). It assigns weaker penalties to rejected tokens whose gradients align with preferred behavior, while maintaining stronger penalties for conflicting tokens. Across 11 benchmarks in mathematics, reasoning, coding, and question answering, GAW‑PO outperforms standard DPO and other baselines, and remains robust when the DPO regularization parameter is reduced.

By Andreea Dutulescu, Stefan Ruseti, Mihai Masala, Traian Rebedea, Mihai Dascalu
arXiv AI
Sep 1

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment

The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.

By Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani