arXiv Machine Learning
DualHeads: Reward-tilted few-step diffusion made simple
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 7
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
The paper introduces Diffusion LAIR, a listwise preference optimization technique that leverages continuous reward scores instead of binary pairwise comparisons to align text‑to‑image diffusion models. LAIR transforms reward scores into centered advantage weights and optimizes an advantage‑weighted regression objective on an implicit reward defined by denoising‑loss improvement over a reference model, with a quadratic penalty to regulate reward magnitude. Experiments demonstrate that Diffusion LAIR surpasses strong baseline methods on SD1.5 and SDXL across generation, compositional, and editing tasks.
arXiv AI
3d ago
DiffusionRVGE: Reinforcement Learning via Regression-Based Value Gradient Estimation
DiffusionRVGE presents a reinforcement learning approach that improves diffusion models by estimating value gradients through regression of centered rewards on centered final denoised images, avoiding costly reward derivatives and backpropagation through the sampling trajectory. The method adapts rollout bandwidth and query weights based on reward continuity, enabling efficient training across multiple rewards. Experiments show that a single SD3.5‑Medium model trained with DiffusionRVGE matches or surpasses DiffusionNFT on most metrics while using roughly one‑fifth of the training compute, and achieves faster reward convergence in single‑reward settings.
arXiv Machine Learning
Oct 12
DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
DiffusionART introduces a new reinforcement learning framework for diffusion models that aggregates prompt rewards into a shared direction, eliminating consistency forces inherent in prior methods. It employs a trust‑region‑masked second update for drift‑adaptive restoration, removing the need for EMA tuning and differentiable reward models. Experiments on image and video generation show that DiffusionART trains more efficiently and stably, achieving higher mean rewards with significantly fewer GPU hours compared to existing approaches.
arXiv Machine Learning
Aug 26
Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
arXiv:2608.23664v1 Announce Type: cross Abstract: Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing method...
arXiv Machine Learning
Jun 2
Drifting Preference Optimization for One-Step Generative Models
arXiv:2606. 02521v1 Announce Type: new Abstract: One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy likelihoods, denoising trajectories, differentiable reward gradients, or test-time optimization.
arXiv Machine Learning
Jun 17
NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment
arXiv:2606. 18066v1 Announce Type: new Abstract: We introduce the Noise-Tilted Reverse Kernel (NTRK), a reward-guided diffusion sampler that injects reward gradients through the noise term, leaving the pretrained reverse kernel unchanged and requiring only a single sample per step.