arXiv AI
Jul 2
PAPA: Online Personalized Active Preference Alignment
arXiv:2607. 00486v1 Announce Type: cross Abstract: Diffusion models are highly effective at modeling complex data distributions, including images and text.
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
The paper introduces Diffusion LAIR, a listwise preference optimization technique that leverages continuous reward scores instead of binary pairwise comparisons to align text‑to‑image diffusion models. LAIR transforms reward scores into centered advantage weights and optimizes an advantage‑weighted regression objective on an implicit reward defined by denoising‑loss improvement over a reference model, with a quadratic penalty to regulate reward magnitude. Experiments demonstrate that Diffusion LAIR surpasses strong baseline methods on SD1.5 and SDXL across generation, compositional, and editing tasks.
Related stories
arXiv Computer Vision
Sep 7
Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generation
The paper introduces Reflection-Aware GRPO (RA‑GRPO), a reinforcement‑learning framework that aligns diffusion generative models with human preferences. It uses Diffusion Reflection to correct intermediate sampling paths by reversing the diffusion process, and Counterfactual Path Synthesis to embed these corrected trajectories into the policy, avoiding extra inference cost. Experiments on text‑to‑image and text‑to‑video models show RA‑GRPO outperforms existing methods, reducing reward hacking and improving generalization while remaining architecture‑agnostic.
arXiv Computer Vision
Oct 8
Personalize at Test Time: Learning User Preferences for Image Generation
arXiv:2610.09015v1 Announce Type: new Abstract: Diffusion models can generate high-quality images, yet aligning their outputs with individual user preferences remains challenging. A key bottleneck is...
arXiv Computer Vision
Sep 10
Multi-dimensional Preference Alignment by Conditioning Reward Itself
arXiv:2512.10237v2 Announce Type: replace Abstract: Reinforcement Learning from Human Feedback has emerged as a standard for aligning diffusion models. However, we identify a fundamental limitation i...
arXiv AI
Sep 30
Diffusion Reward Models
arXiv:2609.33803v2 Announce Type: replace-cross Abstract: Reward models underpin the alignment of large language models, yet the dominant designs reduce each prompt--response pair to a point estimate...
arXiv AI
Oct 5
Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards
The paper presents a post‑training approach for text‑to‑image models that combines a preference reward, trained on large human preference data, with rubric‑based rewards that assess prompt faithfulness and other desirable traits. The authors show that a simple reward composition strategy outperforms a naive weighted average, leading to significant Elo gains on the Arena leaderboard for models like Flux2dev and Ideogram‑4. They also release Arena‑T2I‑Training, a 1K subset of data to aid reproducible research in post‑training.
arXiv AI
Jun 17
STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training
arXiv:2606. 17979v1 Announce Type: new Abstract: Existing RL post-training methods for text-to-image generation usually convert the final-image reward into a single scalar advantage and apply it with the same strength to the entire generative trajectory.
arXiv Computer Vision
Sep 1
Pareto-Guided Optimal Transport for Multi-Reward Alignment
arXiv:2605.13155v2 Announce Type: replace Abstract: Text-to-image generation models have achieved remarkable progress in preference optimization, yet achieving robust alignment across diverse reward...
arXiv Machine Learning
Oct 22
DualHeads: Reward-tilted few-step diffusion made simple
DualHeads is a method that adapts distilled diffusion models, which generate high‑quality images in only a few sampling steps, to align with task requirements and human preferences using only external reward evaluations and reference samples. The approach adds two lightweight binary classifier heads—a reward head that ranks samples from the same prompt and a reference head that distinguishes reference images from generated ones—to a frozen diffusion backbone. By maximizing a weighted sum of the heads’ logits, DualHeads achieves task rewards comparable to 40‑step reinforcement learning methods while maintaining image quality and incurring no additional inference cost.
arXiv Computer Vision
Oct 28
ToPO: Token-Oriented Preference Optimization for Text-to-Image Diffusion Models
ToPO: Token-Oriented Preference Optimization for Text-to-Image Diffusion Models proposes a method that builds a token-conditioned spatial-temporal weighting map for each image pair. By using a frozen reference denoiser to derive spatial and temporal weights from local noise-prediction error differences and linking spatial evidence to content tokens via preferred-branch cross-attention, ToPO reweights Diffusion-DPO and incorporates an auxiliary pixel-midpoint ordering loss. The approach requires no local preference annotations or a separately trained reward model, and achieves state-of-the-art performance with only 500 training updates across multiple preference and compositional evaluations.
arXiv AI
Oct 1
CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models
CAST introduces a reinforcement‑learning fine‑tuning framework for diffusion models that addresses three key limitations: it automatically selects the denoising window based on each model’s trajectory, decomposes prompts into verifiable semantic atoms via Causal Scene Graphs, and applies atom‑level rewards spatially weighted in the policy objective. The method is applied to FLUX.2‑dev and Qwen‑Image‑2512, yielding up to 3.07× improvement on the hardest GenEval 2 prompts compared with Flow‑GRPO while also enhancing overall generation quality.
Hugging Face Trending Papers
Jun 1
Drifting Preference Optimization for One-Step Generative Models
One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy likelihoods, denoising trajectories, differentiable reward gradients, or test-time optimization. We propose Drifting Preference Optimization (DrPO), an online preference-finetuning method for deterministic one-step generators.