arXiv AI
3d ago

Steering Away from Memorization: Reachability-Constrained Reinforcement Learning for Text-to-Image Diffusion

The paper introduces Reachability-Aware Diffusion Steering (RADS), an offline-trained, sampling-time control framework designed to reduce memorization in text-to-image diffusion models while preserving image quality and prompt alignment. RADS treats the diffusion denoising process as a dynamical system and learns a policy-conditioned reachability value function that predicts future memorization risk along the denoising trajectory. By formulating a constrained reinforcement learning problem, the policy steers the trajectory away from memorization through perturbations in the caption embedding space, achieving a better trade‑off between generation diversity, quality, and alignment than existing inference‑time baselines.

By Sathwik Karnik, Juyeop Kim, Sanmi Koyejo, Jong-Seok Lee, Somil Bansal
arXiv AI
Oct 26

PASTA: Prompt-Aware Annealed Diffusion Sampling

PASTA (Prompt-Aware Shared-Latent Training-Free Annealed Diffusion Sampling) is a new framework for diffusion-based image generation that preserves clean, reserved regions such as headlines, logos, or UI elements. It enforces these constraints by selectively applying semantic guidance only to object-allowed areas during sampling, using a shared latent trajectory and a mask-guided fusion of full-prompt and context-prompt guidance. An annealed schedule suppresses semantic leakage early in denoising and gradually restores visual consistency, leading to improved reserved-region fidelity while maintaining overall image realism and perceptual consistency.

By Ren Ajiki, Ryugo Morita, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Andreas Dengel, Jinjia Zhou
arXiv AI
Oct 28

From Structural Feedback to Prompt Policies: Learning Faithful Text-to-Image Prompt Editors

The paper introduces StruPO, a structure‑aware prompt editor that uses a calibrated scene graph to guide prompt modifications while preserving requested scene attributes. By aligning token edits with graph‑based structural feedback, StruPO learns a reusable editing policy that improves faithfulness and relational fidelity across multiple generation backbones. Experiments on diverse test sets and human studies show consistent gains in image quality and preference, especially for complex relations.

By Wentao Ye, Yali Ye, Zhiqing Xiao, Ru Peng, Xinpeng Ti, Liyao Li, Zhanming Shen, Peng Lu, Jie Zhang, Haobo Wang