arXiv Machine Learning
Uncertainty-Normalized Margins for Direct Preference Optimization
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Aug 25
Length-Controlled Margin-Based Preference Optimization without Reference Model
arXiv:2502.14643v3 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF),...
arXiv AI
Jun 12
Boosting Direct Preference Optimization with Penalization
arXiv:2606. 12505v1 Announce Type: cross Abstract: Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset.
arXiv AI
Jul 7
Adaptive Margin RLHF via Preference over Preferences
arXiv:2509. 22851v4 Announce Type: replace-cross Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks.
arXiv AI
3d ago
Optimal Design for Active Preference Learning with Biased LLM Judges
arXiv:2609.38860v1 Announce Type: cross Abstract: Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference lear...
arXiv AI
Jul 29
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.
arXiv AI
Jun 29
Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking
arXiv:2604. 26360v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) systems face a compounding alignment challenge: not only are learned reward models uncertain about unseen state-action pairs, but the human preference annotations they are trained on are themselves inconsistent, context-dependent, and noisy.