arXiv Machine Learning

BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

arXiv:2606. 02947v1 Announce Type: new Abstract: Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks.

arXiv Machine Learning
Aug 14

Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Models

arXiv:2604. 05809v2 Announce Type: replace-cross Abstract: This paper presents Text-Guided Backdoor (TGB), an adjustable backdoor attack against multimodal pretrained models that uses natural-word triggers, namely words that can naturally occur in ordinary textual inputs.

By Yiyang Zhang, Chaojian Yu, Ziming Hong, Yuanjie Shao, Qinmu Peng, Tongliang Liu, Xinge You
arXiv AI
Sep 10

Discovering Natural Transformation Vulnerabilities in Black-Box Vision Models

The paper introduces Adversarial Scenario Attack (ASA), a query‑based black‑box method that discovers natural transformation vulnerabilities in vision models by exploring background, weather, and material/color edits via a multimodal language model and a text‑guided generative editor. ASA outperforms previous query‑based generative attacks on ImageNet classifiers, achieving higher success rates with fewer queries while maintaining perceptual quality. The approach also shows image‑level and prompt‑level transferability, indicating reusable vulnerabilities across models and images.

By Dongsu Song, DaeYun GO, Jay Hoon Jung
Hugging Face Trending Papers
Aug 19

Breaking the weakest link to evade vision language models

The paper investigates how Vision Language Models (VLMs) can be fooled by tiny, human‑imperceptible changes to images. It introduces a gradient‑based attack that targets only the vision encoder, reducing computational cost while still effectively disrupting both untargeted and targeted multimodal interpretations. Experiments on open‑source VLMs such as Qwen2.5‑VL, Granite‑Vision, FastVLM, and Phi‑3.5‑Vision demonstrate that these small perturbations can dramatically alter the models’ textual outputs.