arXiv AI June 12, 2026 By Pengwei Sun

Boosting Direct Preference Optimization with Penalization

arXiv:2606. 12505v1 Announce Type: cross Abstract: Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Aug 25

Length-Controlled Margin-Based Preference Optimization without Reference Model

arXiv:2502.14643v3 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF),...

By Gengxu Li, Tingyu Xia, Yi Chang, Yuan Wu

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 21

Normalized Rewards for Preference Optimization

arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.

By Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, Katherine Metcalf

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 10

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

arXiv:2410. 15595v4 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical.

By Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Zongrui Li, Ruirui Lei, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, Fei Wu

llms reinforcement-learning safety

arXiv Machine Learning

Sep 21

IncentRL: The Trade-Off Between Preference Guidance and Task Performance

arXiv:2609.21525v1 Announce Type: new Abstract: Preference-based reward shaping can guide reinforcement learning, but adding preference signals to the reward may unintentionally change the task being...

By Xuening Wu, Yanlan Kang, Shenqin Yin

reinforcement-learning

arXiv Machine Learning

2d ago

Uncertainty-Normalized Margins for Direct Preference Optimization

arXiv:2609.38647v1 Announce Type: new Abstract: Direct preference optimization (DPO) models binary preferences through a Bradley-Terry model with a common noise scale, without explicitly accounting f...

By Sadegh Khorasani, Petrus Mikkola, Matthias Grossglauser

llms reinforcement-learning

arXiv AI

Jun 11

Autoregressive Direct Preference Optimization

arXiv:2602. 09533v2 Announce Type: replace Abstract: Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences.

By Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

llms reinforcement-learning

One email a morning, machine-written

Related stories

Length-Controlled Margin-Based Preference Optimization without Reference Model

Normalized Rewards for Preference Optimization

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

IncentRL: The Trade-Off Between Preference Guidance and Task Performance

Uncertainty-Normalized Margins for Direct Preference Optimization

Autoregressive Direct Preference Optimization