arXiv Machine Learning August 13, 2026 By Mehryar Mohri, Yutao Zhong

Mind the Gap: Structure-Aware Consistency in Preference Learning

Read the original on arXiv Machine Learning →

arXiv:2604. 27733v2 Announce Type: replace Abstract: Aligning Large Language Models (LLMs) with human intent, whether through explicit reward modeling or direct methods such as DPO, fundamentally relies on minimizing a surrogate loss as a proxy for the true pairwise ranking objective.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 2

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.

By Hyung Gyu Rho

llms nlp reinforcement-learning

arXiv AI

Jun 11

Autoregressive Direct Preference Optimization

arXiv:2602. 09533v2 Announce Type: replace Abstract: Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences.

By Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

llms reinforcement-learning

arXiv AI

Jun 19

AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Models

arXiv:2509. 25148v2 Announce Type: replace Abstract: Post-training alignment of large language models often combines supervised fine-tuning (SFT) on expert demonstrations with reinforcement learning (RL) from preference or verifiable feedback.

By Faqiang Qian, Kang An, Weikun Zhang, Ziliang Wang, Xuhui Zheng, Liangjian Wen, Yong Dai, Mengya Gao, Yichao Wu

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Sep 10

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

arXiv:2609.06893v1 Announce Type: cross Abstract: Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational eff...

By Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi

llms reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Sep 1

Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

arXiv:2608.31079v1 Announce Type: new Abstract: Sycophantic agreement refers to a behavior in which language models excessively affirm the user, often at the cost of factual accuracy. Although sycoph...

By Camila Blank, Zhuofan Ying, Christopher Potts, Peter Hase, Jing Huang

llms reinforcement-learning safety

arXiv AI

Jul 20

RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

arXiv:2605. 04539v4 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO), the efficient alternative to PPO-based RLHF, falls short on knowledge-intensive generation: standard preference signals from human annotators or LLM judges exhibit a systematic verbosity bias that rewards fluency over logical correctness.

By Qiming Bao, Juho Leinonen, Paul Denny, Michael J. Witbrock

llms reinforcement-learning fine-tuning safety

One email a morning, machine-written

Related stories

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

Autoregressive Direct Preference Optimization

AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Models

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization