arXiv:2509. 23102v4 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences.
By Fang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang, Yijia Xiao, Guancheng Wan, Xiaomin Li, Bing Hu, Peng Xia, Jure Leskovec, Yejin Choi
arXiv:2607. 10601v1 Announce Type: new Abstract: Large Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation.
By Yixiong Chen, Alan Yuille
arXiv:2606. 01382v1 Announce Type: cross Abstract: Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward.
By Tianlong Nan, Xiaopeng Li, Christian Kroer, Tianyi Lin
arXiv:2610.15008v1 Announce Type: new
Abstract: Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train polici...
By Riccardo Formenti, Aditya Agrawal, Vikas Garg
The paper investigates how large language models (LLMs) can be trained to cooperate in multi‑agent social dilemmas. It shows that standard reinforcement learning often leads LLM agents to defect and exploit others, even when they have cooperative priors. By adapting the Advantage Alignment algorithm and introducing a group‑relative baseline, the authors fine‑tune LLMs to achieve higher collective payoffs and resist exploitation, and they present a new Trust‑and‑Split environment that requires natural language communication for optimal outcomes.
By Dereck Piche, Mohammed Muqeeth, Milad Aghajohari, Juan Duque, Michael Noukhovitch, Aaron Courville
The paper introduces PARED, an approach that uses inverse reinforcement learning to derive an explicit, interpretable reward from expert demonstrations. By training a lightweight discriminator on response-level features, PARED can guide both inference-time reranking and on-policy reinforcement learning, improving alignment for models that have not undergone supervised fine-tuning and further refining those that have. The method also supports contextual alignment, enabling a single policy to adapt its responses to different audience preferences.
By Micha{\l} Wili\'nski, Liu Leqi, Chirag Nagpal