arXiv Computer Vision

HyperErase: Scale-Calibrated Hypernetwork for Multi-Concept Erasure in Text-to-Image Models

HyperErase introduces a hypernetwork-based framework for concept erasure in text-to-image models, replacing static adapters with prompt-conditioned parameter synthesis. The method maps textual descriptions to LoRA updates, eliminating per-prompt gradient optimization and manual merging. A decoupled rectification strategy further stabilizes and refines the synthesized adapters, yielding improved erasure effectiveness, image quality, and semantic alignment across diverse concepts.

arXiv Computation and Language
Nov 4

ODC: Orthogonal Drift Correction for Improved Text-to-Image Semantic Alignment at Inference

The paper introduces Orthogonal Drift Correction (ODC), an inference-time guidance method that mitigates semantic drift in text-to-image generation without retraining or extra user input. ODC operates in two stages: first it detects drift by comparing the generated image to the prompt in a shared vision-language embedding space, isolates the orthogonal component of the drift, and converts it back into text; second it refines the text conditioning for a second generation pass using an adaptive rank-reduced concept removal module. Experiments show ODC improves prompt-image alignment, raising CLIP scores by up to 5.4% on COALBench-300 and PartiPrompts, and boosting the 3-in-1 score by 4.5% on T2I-CompBench’s complex subset.

By Kasra Kamali, Basim Azam, Tuan Duong Trinh, Jean Honorio, Naveed Akhtar
arXiv Computer Vision
Sep 14

GRACE: Adaptive Concept Erasure with Geometry-Guided Retention in Diffusion Models

GRACE is a new framework for concept erasure in text-to-image diffusion models that uses a semantically weighted sensitive subspace to guide localized interventions and lightweight subspace-constrained adapters to avoid global semantic disruption. It replaces manual counterfactual prompts with an automatically decoupled safe-anchor mechanism and controls intervention strength through an energy-driven dynamic gating system. Experiments show GRACE improves NSFW reduction by 17.86% over five state-of-the-art methods while also reducing target CLIP Score and FID, indicating stronger concept suppression with better preservation of generative quality.

By Qinghui Gong, Yihuai Liang, Yuanlun Xie, Deepak Kumar Jain, Vitomir \v{S}truc, Zhengchun Zhou
arXiv Machine Learning
Jun 2

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.

By Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang, Da-Wei Zhou
arXiv AI
Sep 28

Prompt-Based Continual Compositional Zero-Shot Learning

The paper introduces PromptCCZSL, a framework that enables vision‑language models to continually learn new attributes, objects, and their unique compositions while avoiding forgetting. It uses a frozen VLM backbone with prompt‑based techniques, recency‑weighted multi‑teacher distillation, and several loss functions (CAL, OPL, IDL) to maintain prior knowledge and promote diverse, distinct embeddings. Experiments on UT‑Zappos and C‑GQA show significant performance gains over existing VLM‑based and non‑VLM baselines, establishing a new benchmark for continual compositional zero‑shot learning.

By Sauda Maryam, Sara Nadeem, Faisal Qureshi, Mohsen Ali