arXiv AI

RSPO: Regularized Self-Play Alignment of Large Language Models

arXiv AI

Jun 2

S-SPPO: Semantic-Calibrated Self-Play Preference Optimization

arXiv:2606. 01561v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO).

By Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, ZhengXiao He, Xuanzhao Dong, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo, Abolfazl Razi, Brendan Hogan Rappazzo, Anderson Schneider, Yuriy Nevmyvaka

llms reinforcement-learning

arXiv AI

Aug 12

Multiplayer Nash Preference Optimization

arXiv:2509. 23102v4 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences.

By Fang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang, Yijia Xiao, Guancheng Wan, Xiaomin Li, Bing Hu, Peng Xia, Jure Leskovec, Yejin Choi

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 21

Normalized Rewards for Preference Optimization

arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.

By Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, Katherine Metcalf

llms reinforcement-learning benchmarks safety

arXiv AI

Sep 10

Inference-Time Nash Alignment

arXiv:2609.08082v1 Announce Type: new Abstract: Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to th...

By Hadi Hosseini, Debmalya Mandal, Duohan Zhang

reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jul 7

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

arXiv:2506. 07468v4 Announce Type: replace Abstract: Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities.

By Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, Natasha Jaques

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 2

Efficient Exploration for Iterative Nash Preference Optimization

arXiv:2606. 01382v1 Announce Type: cross Abstract: Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward.

By Tianlong Nan, Xiaopeng Li, Christian Kroer, Tianyi Lin

llms fine-tuning benchmarks safety

arXiv AI

Jun 11

Toward Preference-aligned Large Language Models via Residual-based Model Steering

arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.

By Lucio La Cava, Andrea Tagarelli

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 7

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

arXiv:2512. 15765v3 Announce Type: replace Abstract: Data valuation is a natural framework for understanding which preference datasets matter most when aligning a Large Language Model (LLM) using multiple sources.

By M\'elissa Tamine, Otmane Sakhi, Benjamin Heymann, Maxime Vono, Patrick Loiseau

llms reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jul 30

Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning

arXiv:2607. 26358v1 Announce Type: new Abstract: Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy.

By Keegan Harris, Brian W. Lee, Ian Waudby-Smith, Philip Amortila, Nika Haghtalab, Michael I. Jordan

llms agents reinforcement-learning fine-tuning

arXiv Machine Learning

Jun 9

Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

arXiv:2602. 01357v2 Announce Type: replace Abstract: Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without preference data.

By Shangzhe Li, Xuchao Zhang, Chetan Bansal, Weitong Zhang

llms reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jul 30

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

arXiv:2607. 26094v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models.

By Yunpeng Chu

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 15

Matilda: Engine-Agnostic Search with Human Policy Guidance

arXiv:2606. 25176v2 Announce Type: replace Abstract: Chess engines have evolved from search-based systems optimized solely for strength to neural policies capable of modeling human decisions across much of the rating spectrum.

By Jason Carlson

llms rag benchmarks

Related stories