arXiv AI
Jul 7
Adaptive Margin RLHF via Preference over Preferences
arXiv:2509. 22851v4 Announce Type: replace-cross Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks.
A Framework for Designing Reward Functions: From Objectives to Features to Human-Aligned Reward Functions
arXiv:2608. 12302v1 Announce Type: new Abstract: We present a formal process to enable non-experts to instantiate and iterate on human-aligned reward functions, i.
Related stories
arXiv AI
Jun 2
Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models
arXiv:2604. 17415v3 Announce Type: replace-cross Abstract: Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model.
arXiv AI
Jul 24
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
arXiv:2604. 18239v4 Announce Type: replace-cross Abstract: Preference optimization is widely used to align large language models (LLMs) with human preferences.
arXiv AI
Jun 2
Value-Free Policy Optimization via Reward Partitioning
arXiv:2506. 13702v4 Announce Type: replace-cross Abstract: Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback.
arXiv AI
Aug 3
Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
arXiv:2607. 29246v1 Announce Type: new Abstract: Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases.
arXiv Machine Learning
Jun 30
REAR: Test-time Preference Realignment through Reward Decomposition
arXiv:2606. 30339v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task.
arXiv AI
1d ago
Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
arXiv:2608. 16072v1 Announce Type: cross Abstract: Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners.
arXiv Machine Learning
Jun 18
The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL
arXiv:2606. 19162v1 Announce Type: new Abstract: Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself.
arXiv AI
Jun 30
Distributionally Robust Reinforcement Learning with Human Feedback
arXiv:2503. 00539v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs).
arXiv AI
Jun 6
Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces
arXiv:2606. 05464v1 Announce Type: new Abstract: Verifiable reward training has improved mathematical and coding reasoning, but these domains capture only part of step-by-step decision making.
arXiv Machine Learning
Jul 14
Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability
arXiv:2607. 11432v1 Announce Type: new Abstract: In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert.
arXiv Machine Learning
Aug 4
PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
arXiv:2506. 13741v2 Announce Type: replace-cross Abstract: Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions.