arXiv AI By Haoyu Zhang, Yi Feng, Hanwen Liu, Shibo Zheng, Zhuoxi Wang, Yang Chen, Haowen Xu, Xiangchen Guan, Mohammad Zandsalimy, Shanu Sushmita

The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties

Read the original on arXiv AI →

The paper demonstrates that vision‑language models’ refusal behavior is heavily influenced by whether an image is attached to a request, even when the image is blank or unreadable. Attaching such an image shifts refusal scores by large margins for borderline‑benign prompts while leaving genuinely neutral instructions largely unchanged. This effect varies with image properties, persists across checkpoints, and is not mitigated by explicit instructions to ignore the image.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 11

Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks

arXiv:2608. 01043v2 Announce Type: replace-cross Abstract: We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR).

By Haoyu Zhang, Xiangchen Guan, Shibo Zheng, Mohammad Zandsalimy, Shanu Sushmita
arXiv AI
Aug 11

Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks

arXiv:2607. 26574v2 Announce Type: replace-cross Abstract: Safety classifiers ("guards") are the dominant black-box defense for vision-language models, yet a guard judges an input's surface form, not its meaning: a harmful request re-encoded as set theory, formal logic, a classical language, code, or text rendered inside an image slips past a guard that would block it in plain language - the decode gap.

By Haoyu Zhang, Zhuoxi Wang, Shibo Zheng, Yi Feng, Xiao Luo, Zijian Xiao, Haowen Xu, Xiangchen Guan, Mohammad Zandsalimy, Shanu Sushmita
arXiv AI
Oct 27

Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models

Vision–language models can correctly diagnose medical questions using only the clinical vignette, without the diagnostic image. In experiments on 60 Italian State Examination items, models defended diagnoses derived from a blank placeholder as firmly as those derived from real images, showing no increased fragility. While different adversarial pressures produced distinct dynamics—one leading to a single decisive flip and the other to continual changes—the instability under contentless doubt reflected item ambiguity rather than lack of evidence.

By Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti, Olivia Riccomi, Michele Ferramola, Vittorio De Vita, Alessandro Tosi, Antonio Cristiano, Alessia Longo, Lorenzo De Mori, Chiara Battipaglia, Melissa Sawaya, Luigi De Angelis, Marcello Di Pumpo, Alessandra Piscitelli, Pietro Eric Risuleo, Giulia Vojvodic, Mariapia Vassalli, Nicolo Scarsi, Manuel Del Medico
arXiv AI
Oct 7

Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreaks

The study investigates which image properties enable jailbreak attacks on multimodal models by analyzing four attack families across 313 prompts and five models, including an appendix evaluation of InternVL3.5-8B. It finds that attack images significantly increase success rates compared to benign images, while factors such as tile entropy, JPEG size, and tile count do not reliably predict attack tiles. Only the E4 manipulation, which removes query-specific relatedness, shows a measurable effect on attack success rates, though overall conclusions depend on the rubric judge used.

By Boyuan Chen, Yehia Dawoud, Hailemariam Mersha, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique