Think Before You Score: Thinking Reward Model for Visual Generation
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
RubricRM introduces a pairwise generative reward modeling framework that generates an input‑specific rubric—comprising evaluation dimensions, weights, and scoring criteria—to score candidate images. The method is trained in two stages: supervised fine‑tuning to learn the rubric‑based scoring paradigm and GRPO to refine dimension‑level rewards. Experiments on text‑to‑image generation and instruction‑based image editing benchmarks demonstrate that RubricRM outperforms existing specialized reward models and competes with strong proprietary MLLM judges while using smaller backbones.
The paper introduces a post‑training framework for visual generative models that enhances specific attributes while preserving overall quality. It combines Generator‑Aligned Reward Modeling (GARM) to align reward signals with the generator’s distribution and reduce content confounding, with Target‑Aware Bounded Optimization (TABO) that caps reward values using real target examples. The authors also present TA‑GRPO, a flow‑based video optimization variant that further improves attribute enhancement and mitigates unintended attribute leakage.
The paper examines how current reward models used in text-to-image systems struggle with visual reasoning tasks. By introducing a data‑centric approach that flips the pairing axis—using visual reasoning datasets as preference pairs with the same image but different prompts—the authors achieve simultaneous gains in visual understanding and preference accuracy. This method yields measurable improvements in generation quality on compositional prompts, outperforming a larger baseline model.
The paper presents a post‑training approach for text‑to‑image models that combines a preference reward, trained on large human preference data, with rubric‑based rewards that assess prompt faithfulness and other desirable traits. The authors show that a simple reward composition strategy outperforms a naive weighted average, leading to significant Elo gains on the Arena leaderboard for models like Flux2dev and Ideogram‑4. They also release Arena‑T2I‑Training, a 1K subset of data to aid reproducible research in post‑training.
The paper introduces Objective-aware Trajectory Credit Assignment (OTCA), a framework that refines reinforcement learning for diffusion-based visual generation. OTCA decomposes credit across denoising steps and allocates multiple reward signals adaptively, addressing the coarse, uniform reward assignment of existing GRPO pipelines. Experiments demonstrate that OTCA consistently enhances image and video generation quality across various metrics.
arXiv:2602. 07533v2 Announce Type: replace Abstract: Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models.