arXiv Machine Learning
Jun 8
Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
arXiv:2505. 10892v2 Announce Type: replace Abstract: Post-training LLMs with RLHF and preference optimization methods (e.
A Better Spur Should Start From Each Objective
Related stories
arXiv AI
Aug 3
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
arXiv:2607. 29559v1 Announce Type: new Abstract: Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function.
arXiv AI
Jul 13
Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization
arXiv:2602. 07764v2 Announce Type: replace-cross Abstract: Multi-objective reinforcement learning (MORL) seeks to train agents capable of balancing conflicting objectives.
arXiv Computation and Language
Sep 18
UniPolicy: Unified Objective-Specific Policies for Generative Search Advertising
UniPolicy is a unified objective‑specific policy framework for search advertising that jointly optimizes relevance, click propensity, and commercial value. It uses objective‑aware prefix tokens, sparse MoE‑LoRA routing, and residual FFNs to decouple parameters within a shared backbone, and constructs pairwise preferences from multi‑stage behavioral feedback to strengthen clicked candidates. In large‑scale offline tests and a 7‑day online A/B test, UniPolicy improves CTR by 0.71%, RPS by 1.58%, and advertising revenue by 1.32% while keeping serving latency stable.
arXiv AI
Jun 2
Value-Free Policy Optimization via Reward Partitioning
arXiv:2506. 13702v4 Announce Type: replace-cross Abstract: Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback.
Hugging Face Trending Papers
Sep 17
UniPolicy: Unified Objective-Specific Policies for Generative Search Advertising
UniPolicy is a multi-policy alignment framework for search advertising that jointly optimizes relevance, click propensity, and commercial value. It uses objective-specific prefix tokens, sparse MoE-LoRA routing, and residual FFNs to decouple parameters within a shared backbone, and builds pairwise preferences from multi-stage behavioral feedback to improve generation. In large-scale offline tests and a 7‑day online A/B test, UniPolicy achieved balanced gains across metrics, boosting CTR by 0.71%, RPS by 1.58%, and revenue by 1.32% while keeping latency stable.
arXiv Computer Vision
Sep 10
Multi-dimensional Preference Alignment by Conditioning Reward Itself
arXiv:2512.10237v2 Announce Type: replace Abstract: Reinforcement Learning from Human Feedback has emerged as a standard for aligning diffusion models. However, we identify a fundamental limitation i...
arXiv AI
Jun 4
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
arXiv:2606. 04807v1 Announce Type: new Abstract: Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape.
arXiv Machine Learning
Sep 3
Objective-Behavior Alignment: Diagnostics for MORL Policy Selection
The paper introduces a diagnostic workflow for multi‑objective reinforcement learning (MORL) that reveals behavioral differences among policies on the Pareto front, which are not apparent from value vectors alone. It offers quantitative and visual tools to inspect these variations and demonstrates their effectiveness on both simple grid tasks and more complex continuous‑control benchmarks.
arXiv Computation and Language
Sep 15
Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation
arXiv:2609.14648v1 Announce Type: new Abstract: Aligning multi-turn dialogue agents is usually framed as matching turn-level human preferences, yet direct optimization of long-term outcomes is often...
arXiv Machine Learning
Aug 11
Multi-Agent Reinforcement Learning via Agent-Specific Preference
arXiv:2608. 08604v1 Announce Type: new Abstract: Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions.
arXiv Machine Learning
Oct 28
Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning
The paper introduces Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), a framework that leverages the structure of multiple objectives to enhance cooperation among agents. By formulating MOMARL at both team and individual levels and using objective-wise counterfactual credit assignment, MIRL allows agents to learn how to prioritize objectives individually while jointly optimizing a system-level objective. Experiments on a multi-objective particle environment demonstrate that MIRL, especially its dynamic variant, improves Pareto-front quality compared to baseline methods.