arXiv AI
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 8
Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
arXiv:2505. 10892v2 Announce Type: replace Abstract: Post-training LLMs with RLHF and preference optimization methods (e.
arXiv Machine Learning
Aug 14
Statistical Properties of Robust Learning under Distributional Shifts
arXiv:2608. 13133v1 Announce Type: cross Abstract: Distributional shifts arise when the target deployment environment differs from the source environment that generated the training data.
arXiv Machine Learning
Sep 29
Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment
The paper introduces Geometric Anchor Preference Optimization (GAPO), a method that replaces the static reference policy in Direct Preference Optimization with a dynamic, geometry-aware anchor—a small adversarial perturbation of the current policy. GAPO uses this anchor to adaptively reweight preference pairs based on local sensitivity, and defines an Anchor Gap that approximates worst‑case local margin degradation. Experiments show that GAPO improves robustness to noisy supervision while matching or surpassing existing LLM alignment and reasoning benchmarks.
arXiv Machine Learning
Oct 15
Unification and Optimization of Robust Supervised Learning
arXiv:2605.28165v2 Announce Type: replace Abstract: Robust learning methods such as distributionally robust optimization, vicinal risk minimization, Mixup, and label smoothing address different sourc...
arXiv Machine Learning
Jul 10
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
arXiv:2605. 00155v3 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) is a central post-training tool for aligning large language models, but its training reward is only a learned proxy for true human utility.
arXiv AI
Oct 2
Robust Nash Alignment under Preference Uncertainty
Robust Nash Alignment introduces a game-theoretic framework that seeks a policy with a high worst-case win rate against both an adversarial competitor and any preference kernel within an ambiguity set around a nominal preference. The authors propose a four-player primal-dual proxy game and an optimistic mirror descent-ascent algorithm to efficiently optimize this robust objective, proving convergence guarantees and demonstrating improved performance in controlled tabular games and LLM alignment experiments.