arXiv Computer Vision

Overpainting: Localized Context-aware Diffusion Image Editing

The paper introduces "overpainting," a localized, context-aware image editing technique that allows users to specify precise or loose editing regions via a trimap. The method adapts a pretrained diffusion model with joint attention and low‑rank adaptation, incorporating attention‑dropout to balance noise, source, and mask inputs. An automated pipeline generates training data by pairing images from language‑based editing models, curating them, and extracting trimaps, enabling the model to perform a wide range of editing tasks.

arXiv AI
Oct 15

QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing

QK-Edit introduces a training‑free query‑key manipulation strategy for Multimodal Diffusion Transformers (MM‑DiTs), enabling precise text‑guided image and video editing. By isolating and adjusting critical attention components, it balances prompt fidelity with structural consistency, supporting tasks such as object addition, removal, replacement, background changes, material and color adjustments, and style transformation. The method works seamlessly with state‑of‑the‑art models like FLUX and HunyuanVideo, improving editing performance without requiring additional training.

By Tiancheng Shen, Zilong Huang, Xiangtai Li, Zhijie Lin, Jiyang Liu, Yitong Wang, Jiashi Feng, Ming-Hsuan Yang, Jun Hao Liew
arXiv AI
Oct 2

Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength

Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength introduces SoftPaint, a zero‑shot sampling method that uses soft masks to provide continuous, pixel‑level control over edits in diffusion models. The approach employs a Langevin‑iteration sampler that respects per‑pixel mask strengths, enabling smooth edits from preserving to fully re‑synthesizing content across image and video backbones. SoftPaint is gradient‑free, memory‑efficient, and works universally with existing diffusion models.

By Candi Zheng, Yuan Lan
arXiv AI
Oct 5

VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models

VIDiff is a unified foundation model that uses diffusion techniques to perform a broad range of video tasks, including both understanding tasks like language‑guided video object segmentation and generative tasks such as video editing and enhancement. Unlike prior methods that focus on short clips and require time‑consuming tuning, VIDiff can edit and translate videos within seconds based on user instructions and employs an iterative auto‑regressive approach to maintain consistency in long‑form videos. The authors demonstrate convincing generative results across diverse input videos and written instructions, supported by qualitative and quantitative evidence.

By Zhen Xing, Shuyuan Tu, Qi Dai, Zihao Zhang, Hui Zhang, Han Hu, Zuxuan Wu, Yu-Gang Jiang
arXiv Computer Vision
Oct 19

MTEdit: Packed Trajectories Beat Rollouts for Multi-Turn Image Editing

MTEdit is a new multi‑turn image editing dataset that uses packed trajectory canvas (PTC) generation to create 500K editing trajectories, each up to eight turns with an average of 5.38 turns. The dataset addresses visual‑quality drift by jointly generating multiple editing states within a unified visual context, covering both step‑wise and context‑aware editing scenarios. MTEdit-Bench evaluates models on instruction following, content consistency, visual‑quality drift resistance, and historical context usage, and fine‑tuned Qwen‑Image‑MTEdit outperforms other open‑source models on this benchmark.

By Houyuan Chen, Fanzhou Wang, Xiangyu Fan, Songlin Yang, Zhitao Yang, Lvmin Zhang, Pei Yang, Hao Zhao, Lei Yang, Anyi Rao