arXiv AI
Jul 21
Normalized Rewards for Preference Optimization
arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.
Boosting Direct Preference Optimization with Penalization
arXiv:2606. 12505v1 Announce Type: cross Abstract: Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset.
Related stories
arXiv AI
Jun 10
A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
arXiv:2410. 15595v4 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical.
arXiv AI
Jun 11
Autoregressive Direct Preference Optimization
arXiv:2602. 09533v2 Announce Type: replace Abstract: Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences.
arXiv AI
Jun 2
Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.
arXiv AI
Jul 29
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.
arXiv AI
Jul 7
Adaptive Margin RLHF via Preference over Preferences
arXiv:2509. 22851v4 Announce Type: replace-cross Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks.
arXiv Machine Learning
Jul 30
Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
arXiv:2607. 26094v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models.
arXiv AI
Jul 24
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
arXiv:2604. 18239v4 Announce Type: replace-cross Abstract: Preference optimization is widely used to align large language models (LLMs) with human preferences.
arXiv AI
Jun 19
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
arXiv:2606. 19744v1 Announce Type: cross Abstract: Aligning language models with human preferences often requires optimising multiple behavioural objectives.
arXiv AI
Jun 2
Efficient Exploration for Iterative Nash Preference Optimization
arXiv:2606. 01382v1 Announce Type: cross Abstract: Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward.
arXiv AI
Jun 9
DynamicPO: Dynamic Preference Optimization for Recommendation
arXiv:2605. 00327v2 Announce Type: replace-cross Abstract: In large language model (LLM)-based recommendation systems, direct preference optimization (DPO) effectively aligns recommendations with user preferences, requiring multi-negative objective functions to leverage abundant implicit-feedback negatives and sharpen preference boundaries.
arXiv AI
Jun 24
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
arXiv:2509. 03647v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly serve as automated evaluators, yet they suffer from "self-preference bias": a tendency to favor their own outputs over those of other models.