arXiv AI

Which Pairs to Compare for LLM Post-Training?

arXiv:2606. 19607v1 Announce Type: new Abstract: Preference-based post-training has become a central paradigm for aligning language models.

arXiv Machine Learning

22h ago

Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training

arXiv:2608. 16926v1 Announce Type: new Abstract: Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance.

By Peng Sun, Yi Yang, Antong Zhang, Chunxiao Li, Yanbo Wang, Dianbo Liu, xin chen, Kai Yu, Lu Chen, Tianfan Fu

llms reinforcement-learning fine-tuning

arXiv AI

Jun 2

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.

By Hyung Gyu Rho

llms nlp reinforcement-learning

arXiv AI

Jun 10

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

arXiv:2410. 15595v4 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical.

By Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Zongrui Li, Ruirui Lei, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, Fei Wu

llms reinforcement-learning safety

arXiv AI

Jun 11

Toward Preference-aligned Large Language Models via Residual-based Model Steering

arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.

By Lucio La Cava, Andrea Tagarelli

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 14

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

arXiv:2607. 09796v1 Announce Type: new Abstract: Direct Preference Optimization (DPO) has become an important method for aligning large language models (LLMs) with human preferences because it removes the need for explicit reward modeling and reinforcement learning optimization.

By Hua Qu, Yifan Li, Xiaodong Yuan

llms nlp reinforcement-learning fine-tuning safety

arXiv AI

Jun 11

Autoregressive Direct Preference Optimization

arXiv:2602. 09533v2 Announce Type: replace Abstract: Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences.

By Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

llms reinforcement-learning

arXiv AI

Jul 13

Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys

arXiv:2604. 17267v2 Announce Type: replace Abstract: Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions.

By Zikun Ye, Hema Yoganarasimhan

llms

arXiv AI

Jul 29

Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization

arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.

By Zhengtao Yao, Runhao Li, Xupeng Chen, Jiayi Cheng, Chenqian Le, Michael Yue, Siheng Wang, Haoyan Xu, Yuqi Li, Chenhao Wei, Zhengdao Li, Rongchao Zhang, Guang Yang, Yidong Wang, Junhao Dong

llms safety

arXiv AI

Jun 2

S-SPPO: Semantic-Calibrated Self-Play Preference Optimization

arXiv:2606. 01561v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO).

By Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, ZhengXiao He, Xuanzhao Dong, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo, Abolfazl Razi, Brendan Hogan Rappazzo, Anderson Schneider, Yuriy Nevmyvaka

llms reinforcement-learning

arXiv Machine Learning

Jul 31

AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization

arXiv:2607. 27953v1 Announce Type: new Abstract: Combinatorial optimization problems (COPs) underpin many real-world decisions, but their exponentially large search spaces make high-quality solutions costly to obtain.

By Shengda Gu, Kai Li, Xinyi Ke, Haobo Fu, Yifan Zhang, Jian Cheng

llms reinforcement-learning

arXiv AI

Jul 21

Normalized Rewards for Preference Optimization

arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.

By Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, Katherine Metcalf

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 19

Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings

arXiv:2606. 19744v1 Announce Type: cross Abstract: Aligning language models with human preferences often requires optimising multiple behavioural objectives.

By Pranav Bhandari, Nicolas Fay, Amitava Datta, Usman Naseem, Mehwish Nasim

llms reinforcement-learning fine-tuning safety

Related stories