arXiv:2606. 01382v1 Announce Type: cross Abstract: Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward.
arXiv:2602. 02572v2 Announce Type: replace-cross Abstract: Existing alignment methods directly use the reward model learned from user preference data to optimize an LLM policy, subject to KL regularization with respect to the base policy.
The paper introduces Stackelberg Alignment, a leader‑follower framework that lets a pool of language models collaborate and improve by learning from each other’s responses. An EXP3 bandit leader adaptively selects instructions based on difficulty and discriminability, while the models act as followers, evaluating peers and learning via DPO or GRPO with Elo‑style reputation weighting and opponent matching. Experiments on diverse benchmarks show that this adaptive curriculum outperforms static baselines by up to 12‑25% and improves multi‑LLM evolution.
Robust Nash Alignment introduces a game-theoretic framework that seeks a policy with a high worst-case win rate against both an adversarial competitor and any preference kernel within an ambiguity set around a nominal preference. The authors propose a four-player primal-dual proxy game and an optimistic mirror descent-ascent algorithm to efficiently optimize this robust objective, proving convergence guarantees and demonstrating improved performance in controlled tabular games and LLM alignment experiments.
arXiv:2606. 09124v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals.
arXiv:2509. 23102v4 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences.
arXiv:2601.08777v2 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) to serve users with heterogeneous and potentially conflicting preferences is a central challenge for pe...
The paper introduces a doubly robust preference optimization algorithm for reinforcement learning from human feedback (RLHF) in large language models. It addresses sensitivity to misspecifications in the preference model, reference policy, or reward function by ensuring consistency when either the preference model or the reference policy is correctly specified. Experiments show that the method outperforms state‑of‑the‑art algorithms both theoretically and empirically.
arXiv:2609.08082v1 Announce Type: new Abstract: Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to th...
arXiv:2607. 03248v1 Announce Type: cross Abstract: The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization.
arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.
arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.