arXiv Computer Vision

Rethinking Memorization Mitigation in Diffusion Models: Reinforcing Text Conditioning

arXiv AI
3d ago

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

The paper introduces Reachability-Aware Diffusion Steering (RADS), an offline-trained, sampling-time control framework designed to reduce memorization in text-to-image diffusion models while preserving image quality and prompt alignment. RADS treats the diffusion denoising process as a dynamical system and learns a policy-conditioned reachability value function that predicts future memorization risk along the denoising trajectory. By formulating a constrained reinforcement learning problem, the policy steers the trajectory away from memorization through perturbations in the caption embedding space, achieving a better trade‑off between generation diversity, quality, and alignment than existing inference‑time baselines.

By Sathwik Karnik, Juyeop Kim, Sanmi Koyejo, Jong-Seok Lee, Somil Bansal
arXiv AI
Oct 26

PASTA: Prompt-Aware Annealed Diffusion Sampling

PASTA (Prompt-Aware Shared-Latent Training-Free Annealed Diffusion Sampling) is a new framework for diffusion-based image generation that preserves clean, reserved regions such as headlines, logos, or UI elements. It enforces these constraints by selectively applying semantic guidance only to object-allowed areas during sampling, using a shared latent trajectory and a mask-guided fusion of full-prompt and context-prompt guidance. An annealed schedule suppresses semantic leakage early in denoising and gradually restores visual consistency, leading to improved reserved-region fidelity while maintaining overall image realism and perceptual consistency.

By Ren Ajiki, Ryugo Morita, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Andreas Dengel, Jinjia Zhou
arXiv AI
Oct 28

From Structural Feedback to Prompt Policies: Learning Faithful Text-to-Image Prompt Editors

The paper introduces StruPO, a structure‑aware prompt editor that uses a calibrated scene graph to guide prompt modifications while preserving requested scene attributes. By aligning token edits with graph‑based structural feedback, StruPO learns a reusable editing policy that improves faithfulness and relational fidelity across multiple generation backbones. Experiments on diverse test sets and human studies show consistent gains in image quality and preference, especially for complex relations.

By Wentao Ye, Yali Ye, Zhiqing Xiao, Ru Peng, Xinpeng Ti, Liyao Li, Zhanming Shen, Peng Lu, Jie Zhang, Haobo Wang
arXiv Machine Learning
Sep 25

CARE: Condition-Aware Representation Regularization for Diffusion Models

The paper introduces CARE, a lightweight, plug‑and‑play regularization framework for diffusion models that dynamically adjusts feature distributions based on condition similarity. By leveraging built‑in conditioning signals such as labels or text prompts, CARE promotes tighter feature clusters for similar conditions without requiring explicit alignment losses or external supervision. Empirical results show consistent improvements in visual fidelity and convergence stability, achieving significant FID reductions and speed‑ups on ImageNet and text‑to‑image tasks, and it can be combined with existing regularization methods for further gains.

By Fengjia Guo, Zhuoyi Yang, Jie Tang
arXiv AI
Nov 5

ReMoA: Recursive Modular Adaptation via Sparse Latent Refinement in Multimodal Diffusion Models

ReMoA introduces a Recursive Modular Adaptation framework that embeds sparse neural modules into joint attention layers of diffusion models, enabling iterative refinement of visual tokens across multiple recursion steps. The method leverages modular human cognition principles to share parameters efficiently while improving text-to-image generation. Experiments on ImageNet and GenEval/DPG benchmarks show that ReMoA enhances object detail, alignment, and recall without adding extra parameters.

By Yuwei Sun, Yuxuan Yao, Hui Li, Weihao Yuan, Siyu Zhu
arXiv Computer Vision
Sep 7

Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generation

The paper introduces Reflection-Aware GRPO (RA‑GRPO), a reinforcement‑learning framework that aligns diffusion generative models with human preferences. It uses Diffusion Reflection to correct intermediate sampling paths by reversing the diffusion process, and Counterfactual Path Synthesis to embed these corrected trajectories into the policy, avoiding extra inference cost. Experiments on text‑to‑image and text‑to‑video models show RA‑GRPO outperforms existing methods, reducing reward hacking and improving generalization while remaining architecture‑agnostic.

By Junlong Wu, Jiuzhou Lin, Jia Sun, Boheng Zhang, Huaiqing Wang, Dewen Fan, Houde Liu, Qianqian Gan, Fan Yang, Tingting Gao