arXiv AI
Jun 30
Aligning Language Model Benchmarks with Pairwise Preferences
arXiv:2602. 02898v3 Announce Type: replace Abstract: Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance.
REAR: Test-time Preference Realignment through Reward Decomposition
arXiv:2606. 30339v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task.
Related stories
arXiv AI
Jun 11
Toward Preference-aligned Large Language Models via Residual-based Model Steering
arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.
arXiv AI
Sep 1
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
The paper introduces Personalized Group Relative Policy Optimization (P‑GRPO), a new alignment framework for large language models that separates advantage estimation from immediate batch statistics. By normalizing advantages using preference‑group‑specific reward histories instead of the concurrent generation group, P‑GRPO maintains contrastive signals for distinct user preferences. Experiments across various tasks show that P‑GRPO converges faster and yields higher rewards than standard GRPO, improving alignment with heterogeneous human preferences while preserving general capabilities.
arXiv AI
Aug 18
Leveraging Machine Unlearning for Cost-Efficient Preference Alignment
arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.
arXiv AI
Jun 2
T-POP: Test-Time Personalization with Online Preference Feedback
arXiv:2509. 24696v2 Announce Type: replace-cross Abstract: Personalizing large language models (LLMs) to individual user preferences is a critical step beyond generating generically helpful responses.
arXiv AI
2d ago
Gradient-Aligned Pair Selection for Personalized Preference Optimization
The paper introduces GAP-DPO, a method for personalizing large language models by selecting preference pairs based on gradient alignment with user utility. It formalizes personalized preference learning as a geometry‑aligned optimization problem, showing that off‑policy sampling can shift DPO updates from error correction to reinforcement when preference margins align with utility gradients. Experiments demonstrate that GAP‑DPO improves stylistic fidelity, preference alignment, and overall generation quality over standard DPO variants.
arXiv AI
Jul 7
Unbiased Alignment for Large Language Models with Noisy Preferences
arXiv:2607. 03248v1 Announce Type: cross Abstract: The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization.
arXiv Machine Learning
Sep 23
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
arXiv:2504.20106v4 Announce Type: replace Abstract: Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessiv...
arXiv AI
Aug 28
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
The paper introduces Cross‑lingual Ranking Preference Optimization (CRPO), a framework that uses high‑quality English preference data to improve alignment of large language models in other languages. CRPO builds a hierarchical structure over parallel preference pairs, jointly optimizing intra‑ and inter‑lingual preferences and providing a relative ranking signal beyond binary comparisons. Experiments on five languages show consistent gains in instruction‑following and knowledge utilization, with robust performance across different weighting schemes and improved reward margins and log‑probabilities of desirable responses.
arXiv AI
Aug 25
Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization
arXiv:2608.23149v1 Announce Type: cross Abstract: The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in...
arXiv Machine Learning
Jul 31
Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.
arXiv AI
Jun 11
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
arXiv:2605. 12288v3 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions.