← Back to all news
arXiv AI By Cheol Woo Kim, Jai Moondra, Roozbeh Nahavandi, Andrew Perrault, Milind Tambe, Swati Gupta

Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment

Read the original on arXiv AI →

The paper introduces PALM, an algorithm that constructs a compact portfolio of aligned large language models (LLMs) capable of near‑optimal performance across a wide range of reward weightings for objectives like helpfulness, harmlessness, and conciseness. PALM uses a structured grid of weight vectors, lazy search, and pruning to efficiently identify a small set of policies that provably cover all target reward configurations within specified tolerances. Experiments demonstrate that PALM achieves smaller approximation gaps than portfolios built from uniformly spaced or randomly sampled weights and scales effectively to higher‑dimensional reward spaces.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

Hugging Face Trending Papers
Jun 3

BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization

Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates.

llmsreinforcement-learningfine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Jul 31

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.

By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv AI
Sep 1

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment

The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.

By Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani
llmsreinforcement-learningsafety
More like this →
arXiv Machine Learning
Jun 25

MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources

arXiv:2606. 25832v1 Announce Type: new Abstract: Achieving strong optimization generalization across diverse optimization problems while requiring limited training resources remains a challenging problem for optimization-oriented large language models (LLMs).

By Ke Zhao, Zixiang Di, Hong Qian, Xiang Shu, Yaolin Wen, Qitao Shi, Bingdong Li, Xingyu Lu, Xiangfeng Wang, Jun Zhou, Ke Tang, Yang Yu
llmsreinforcement-learning
More like this →