arXiv Machine Learning
Jun 8
Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
arXiv:2505. 10892v2 Announce Type: replace Abstract: Post-training LLMs with RLHF and preference optimization methods (e.
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
Related stories
arXiv Machine Learning
Aug 14
Statistical Properties of Robust Learning under Distributional Shifts
arXiv:2608. 13133v1 Announce Type: cross Abstract: Distributional shifts arise when the target deployment environment differs from the source environment that generated the training data.
arXiv Machine Learning
Sep 29
Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment
The paper introduces Geometric Anchor Preference Optimization (GAPO), a method that replaces the static reference policy in Direct Preference Optimization with a dynamic, geometry-aware anchor—a small adversarial perturbation of the current policy. GAPO uses this anchor to adaptively reweight preference pairs based on local sensitivity, and defines an Anchor Gap that approximates worst‑case local margin degradation. Experiments show that GAPO improves robustness to noisy supervision while matching or surpassing existing LLM alignment and reasoning benchmarks.
arXiv Machine Learning
Oct 15
Unification and Optimization of Robust Supervised Learning
arXiv:2605.28165v2 Announce Type: replace Abstract: Robust learning methods such as distributionally robust optimization, vicinal risk minimization, Mixup, and label smoothing address different sourc...
arXiv Machine Learning
Jul 10
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
arXiv:2605. 00155v3 Announce Type: replace Abstract: Reinforcement learning from human feedback (RLHF) is a central post-training tool for aligning large language models, but its training reward is only a learned proxy for true human utility.
arXiv AI
Oct 2
Robust Nash Alignment under Preference Uncertainty
Robust Nash Alignment introduces a game-theoretic framework that seeks a policy with a high worst-case win rate against both an adversarial competitor and any preference kernel within an ambiguity set around a nominal preference. The authors propose a four-player primal-dual proxy game and an optimistic mirror descent-ascent algorithm to efficiently optimize this robust objective, proving convergence guarantees and demonstrating improved performance in controlled tabular games and LLM alignment experiments.
arXiv Machine Learning
Jul 21
Representation-Aware Distributionally Robust Optimization: A Knowledge Transfer Framework
arXiv:2509. 09371v2 Announce Type: replace-cross Abstract: Distributionally robust optimization (DRO) protects statistical learning against distributional shifts by optimizing the worst-case performance over a set of perturbed distributions.
arXiv Machine Learning
2d ago
Doubly Robust Distributionally Robust Offline Contextual Pricing
The paper introduces a doubly robust framework for distributionally robust offline contextual pricing, addressing challenges of distributional shifts and continuous actions. It proposes a localized DR estimator that is computationally efficient and semiparametrically optimal, and a learning algorithm that exploits demand noise smoothness to achieve finite‑sample regret bounds of τ^{-σ/(2σ+1)} for smoothness orders s=1,2. Experiments across varying distribution shifts demonstrate the effectiveness of the proposed methods.
arXiv Machine Learning
Aug 13
Mind the Gap: Structure-Aware Consistency in Preference Learning
arXiv:2604. 27733v2 Announce Type: replace Abstract: Aligning Large Language Models (LLMs) with human intent, whether through explicit reward modeling or direct methods such as DPO, fundamentally relies on minimizing a surrogate loss as a proxy for the true pairwise ranking objective.
arXiv AI
Jul 3
Distributionally Robust Listwise Preference Optimization
arXiv:2607. 01715v1 Announce Type: new Abstract: Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level.
arXiv AI
Jun 30
Distributionally Robust Reinforcement Learning with Human Feedback
arXiv:2503. 00539v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs).
arXiv Machine Learning
Oct 9
Calibrating Ambiguity Set via Diagnostic Transport for Distributionally Robust Optimization
The paper introduces Diagnostic-Transport DRO (DT‑DRO), a method that calibrates the geometry of the ambiguity set in distributionally robust optimization using held‑out data. By applying the conditional probability integral transform to detect systematic probability misallocation, DT‑DRO adjusts both the ambiguity‑set center and ground cost through an outcome‑level transport, yielding a computationally tractable dual reformulation. The authors provide theoretical guarantees on ambiguity radii and decision‑risk bounds that improve as estimation errors shrink, and demonstrate that DT‑DRO can remove the robustness floor caused by model misspecification in synthetic and power‑outage experiments.