The paper investigates how Vision Language Models (VLMs) can be fooled by tiny, human‑imperceptible changes to images. It introduces a gradient‑based attack that targets only the vision encoder, reducing computational cost while still effectively disrupting both untargeted and targeted multimodal interpretations. Experiments on open‑source VLMs such as Qwen2.5‑VL, Granite‑Vision, FastVLM, and Phi‑3.5‑Vision demonstrate that these small perturbations can dramatically alter the models’ textual outputs.
The paper investigates how Vision Language Models (VLMs) can be fooled by small, human‑imperceptible changes to images. It introduces a gradient‑based attack that targets only the vision encoder, reducing computational cost while still effectively disrupting both untargeted and targeted multimodal alignment. Experiments on open‑source VLMs such as Qwen2.5‑VL, Granite‑Vision, FastVLM, and Phi‑3.5‑Vision demonstrate that these perturbations can significantly alter the models’ textual outputs.
By Ilan Zini, Boussad Addad, Katarzyna Kapusta
arXiv:2604. 05809v2 Announce Type: replace-cross Abstract: This paper presents Text-Guided Backdoor (TGB), an adjustable backdoor attack against multimodal pretrained models that uses natural-word triggers, namely words that can naturally occur in ordinary textual inputs.
By Yiyang Zhang, Chaojian Yu, Ziming Hong, Yuanjie Shao, Qinmu Peng, Tongliang Liu, Xinge You
arXiv:2606. 02947v1 Announce Type: new Abstract: Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks.
By Ivan Saboli\'c, Marin Or\v{s}i\'c, Josip \v{S}ari\'c, Sven Lon\v{c}ari\'c
arXiv:2606. 24388v1 Announce Type: new Abstract: We introduce a large-scale, open-source dataset of pre-generated adversarial attacks for vision-language models (VLMs).
By Simone Gallivanone, Hossein Khodadadi, Mauro Dore, Mauro Medda, Nicola Franco
FusionAudit introduces PaRMA, a pathway-conditioned robustness auditing framework for native multimodal vision‑language models. It evaluates two image‑level interventions—pixel perturbation and rendered‑text insertion—to expose architecture‑specific blind spots. The study applies diagnostic‑motivated attacks (MANTIS and Nat‑Evid) to 18 open‑weight VLMs, revealing that legacy single‑attack rankings are unstable and that PaRMA uncovers higher residual risk than standard benchmarks.
By Jiaming Zhang, Zherui Li, Siqi Guo, Junhao Dong, Kun Wang, Wei Yang Bryan Lim
Unified autoregressive models (UAMs) generate both text and image tokens in a single pass, simplifying multimodal generation but introducing new vulnerabilities. The authors present the Token by Token Backdoor Attack (ToBAC), the first backdoor attack targeting UAMs, which can be executed via data-based or model-based poisoning. ToBAC can manipulate both visual outputs and accompanying text, enabling subtle triggers such as the word "cool" to induce brand promotion or ideological influence in a majority of generations, with success rates up to 63.1% against JanusPro.
By Tobias Braun, Jonas Henry Grebe, Hossein Shakibania, Anna Rohrbach, Marcus Rohrbach
Multi-modal Large Language Models (MLLMs) achieve strong performance on vision-language tasks, but incorporating visual inputs through a vision encoder (e. g.
Recent advancements in Image-to-Video (I2V) generation have transformed input images from simple appearance references into interactive control interfaces where visual cues such as arrows, sketches, and emojis orchestrate complex video dynamics with unprecedented controllability. However, these seemingly innocuous static cues can be interpreted by models as executable temporal instructions, unfolding into harmful actions in the generated videos.
arXiv:2603. 29418v2 Announce Type: replace-cross Abstract: Although multimodal large language models (MLLMs) are increasingly deployed in real-world applications, their instruction-following behavior leaves them vulnerable to prompt injection attacks.
By Meiwen Ding, Song Xia, Chenqi Kong, Xudong Jiang
arXiv:2604.06502v2 Announce Type: replace
Abstract: Vision-Language Models (VLMs) face significant safety vulnerabilities from malicious prompt attacks due to weakened alignment during visual integra...
By Peigui Qi, Kunsheng Tang, Yanpu Yu, Jialin Wu, Yide Song, Wenbo Zhou, Zhicong Huang, Cheng Hong, Weiming Zhang, Nenghai Yu
arXiv:2605.16090v2 Announce Type: replace-cross
Abstract: Large vision-language models (LVLMs) have emerged as a powerful paradigm for multimodal intelligence, but their growing deployment also expan...
By Hao Yang, Zhuo Ma, Yang Liu, Yilong Yang, Guancheng Wang, JianFeng Ma