arXiv:2602.02140v2 Announce Type: replace
Abstract: Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, wheth...
By Chenlong Wang, Yuhang Chen, Zhihan Hu, Dongping Chen, Wenhu Chen, Sarah Wiegreffe, Tianyi Zhou
arXiv:2608. 05000v1 Announce Type: cross Abstract: Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining.
By Junlin Han, Shengbang Tong, David Fan, Minghao Chen, Philip Torr, Filippos Kokkinos, Mike Lewis
The paper introduces UniSandbox, a decoupled evaluation framework with controlled synthetic datasets, to study whether understanding informs generation in Unified Multimodal Models. Results show a notable understanding‑generation gap, especially in reasoning generation and knowledge transfer. Explicit Chain‑of‑Thought (CoT) in the understanding module bridges this gap, and self‑training can internalize CoT for implicit reasoning during generation; query‑based architectures also exhibit latent CoT‑like properties that aid knowledge transfer.
By Yuwei Niu, Weiyang Jin, Jiaqi Liao, Chaoran Feng, Peng Jin, Bin Lin, Zongjian Li, Bin Zhu, Weihao Yu, Li Yuan
The paper investigates how new concepts can be integrated into unified multimodal models (UMMs) by separating generation and understanding objectives through a novel visual entity bound to a single task direction. Experiments show that the effectiveness of cross‑task usability depends on where the concept is injected into the shared computation, with a mid‑stack alignment objective achieving high concept acquisition with minimal loss to overall performance. The study highlights that unified weights alone are insufficient; the two directions must share a semantic format at the entry point for efficient concept integration.
By Zongyang Qiu, Yihan Wu, Kaixuan Fan, Bo Li, Hui Xiong
arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.
By Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, Phil Woodland, Ricard Marxer
UnifiedAttack introduces a benchmark for testing the safety of large multimodal models (LMMs) in tasks that combine text and image to produce harmful content. The benchmark focuses on the additional harm that arises from cross‑modal synergy and includes filtered multimodal samples and synthesized disinformation queries. A synergistic hijacking framework—comprising In‑Context Reskinning (ICR) and Cognitive Planning Injection (CPI)—is proposed to expose vulnerabilities, and extensive evaluations show that UnifiedAttack consistently bypasses current alignment defenses in state‑of‑the‑art architectures.
By Bingjun Luo, Jialin Guo, Tony Wang, Siqi Li