arXiv AI
Jul 21
Normalized Rewards for Preference Optimization
arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.
MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment
arXiv:2608. 14828v1 Announce Type: new Abstract: Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help.
Related stories
arXiv AI
Jul 24
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
arXiv:2604. 18239v4 Announce Type: replace-cross Abstract: Preference optimization is widely used to align large language models (LLMs) with human preferences.
arXiv Machine Learning
Jun 2
The Representation-Rationalizability Tradeoff in Reward Learning
arXiv:2606. 00291v1 Announce Type: cross Abstract: In RLHF, each training example contains a prompt $x$ and two candidate responses $y,y'$, and annotators provide pairwise preferences between these responses.
arXiv Machine Learning
Jun 16
GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
arXiv:2606. 16771v1 Announce Type: new Abstract: As LLMs advance, post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities.
arXiv AI
Aug 12
Multiplayer Nash Preference Optimization
arXiv:2509. 23102v4 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences.
arXiv AI
Jun 2
Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.
arXiv AI
Aug 12
Procedural Fairness Failures in RLHF from Preference Averaging
arXiv:2608. 10126v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity.
arXiv AI
Jul 7
Out-of-Distribution Generalization of Risk Aversion in Language Models
arXiv:2607. 02755v1 Announce Type: cross Abstract: Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned.
arXiv AI
Jul 7
Attention Limited Reward Learning
arXiv:2607. 04590v1 Announce Type: new Abstract: Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences.
arXiv AI
Aug 3
Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing
arXiv:2607. 28814v1 Announce Type: cross Abstract: In Motivational Interviewing (MI), a client's sustain talk (arguments for the status quo) calls for the counselor to roll with resistance, a move that can fail in two opposite ways: capitulation (abandoning the change agenda to preserve rapport) or confrontation (arguing or directing, overriding the client's autonomy).
arXiv AI
Aug 5
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
arXiv:2608. 03092v1 Announce Type: cross Abstract: We aim to improve model performance in multi-reward reinforcement learning training process.
arXiv AI
1d ago
Leveraging Machine Unlearning for Cost-Efficient Preference Alignment
arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.