arXiv AI
Inference-Time Nash Alignment
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 2
Robust Nash Alignment under Preference Uncertainty
Robust Nash Alignment introduces a game-theoretic framework that seeks a policy with a high worst-case win rate against both an adversarial competitor and any preference kernel within an ambiguity set around a nominal preference. The authors propose a four-player primal-dual proxy game and an optimistic mirror descent-ascent algorithm to efficiently optimize this robust objective, proving convergence guarantees and demonstrating improved performance in controlled tabular games and LLM alignment experiments.
arXiv Machine Learning
Jul 7
Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization
arXiv:2512. 15765v3 Announce Type: replace Abstract: Data valuation is a natural framework for understanding which preference datasets matter most when aligning a Large Language Model (LLM) using multiple sources.
arXiv AI
Jun 2
Efficient Exploration for Iterative Nash Preference Optimization
arXiv:2606. 01382v1 Announce Type: cross Abstract: Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward.
arXiv AI
Aug 12
Multiplayer Nash Preference Optimization
arXiv:2509. 23102v4 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences.
arXiv AI
Sep 17
A Zeroth-Order Paradigm for LLM Preference Alignment
The paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order method that aligns large language models with human preferences using comparison oracles instead of direct differentiable loss optimization. It provides theoretical convergence guarantees for both offline and online variants under smoothness, gradient sparsity, and oracle compatibility assumptions, and establishes performance bounds under local coverage and in-distribution reward accuracy. Experiments on several LLMs (Mistral, Llama, Gemma-2, Qwen3, Gemma-3) show that ComPO outperforms existing direct alignment methods, achieving higher length-controlled win rates and diagnostics that suggest mitigation of likelihood displacement.
arXiv AI
Jun 9
Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
arXiv:2602. 02572v2 Announce Type: replace-cross Abstract: Existing alignment methods directly use the reward model learned from user preference data to optimize an LLM policy, subject to KL regularization with respect to the base policy.