Aligned vision‑language models (VLMs) are designed to combine grounded visual reasoning with safe generation. The study finds that when safety constraints are applied, these models often abstain from answering questions that they could answer under default instruction, yet visual evidence continues to influence the decoding process. The authors show that safety‑induced abstention alters late‑stage hidden‑state dynamics, and that targeted interventions can restore grounded answering without retraining or changing visual inputs.
SafeRI proposes an on-demand safety alignment approach for large vision-language models, contrasting with existing always-on methods that globally modify model behavior. The framework uses a lightweight recognizer to evaluate token-level safety during autoregressive generation, gating a LoRA module that only activates when unsafe content is detected. By training the LoRA on unsafe prefixes and safe continuations, SafeRI redirects unsafe generations back to safe responses without perturbing the model’s original reasoning path.
By Caoyuan Ma, Tian Gu, Wenpu Liu, Weichu Xie, Shuai Dong, Yuqi Xu, Ji Zhao, Ziyue Wang, Wenzheng Chang, Taiqiang Wu, Yongfu Zhu, Wenqi Shao, Zheng Wang, Yinqiang Zheng
arXiv:2611.07216v1 Announce Type: cross
Abstract: Large vision-language models (LVLMs) are vulnerable to compositional harm: an image and a prompt that are benign in isolation become harmful through...
By Hong kyu Lee, Kai Shu, Li Xiong
arXiv:2608.30750v1 Announce Type: new
Abstract: Vision-language models (VLMs) can comply with harmful requests delivered through images, even when their LLM backbones would refuse the same content in...
By Jiaxuan Li, Jiahao Zhang, Duc Minh Vo, Huy H. Nguyen, Pride Kavumba, Koki Wataoka
arXiv:2509. 25533v2 Announce Type: replace-cross Abstract: As Vision Language Models (VLMs) are deployed across safety-critical applications, understanding and controlling their behavioral patterns has become increasingly important.
By Ravikumar Balakrishnan, Mansi Phute
arXiv:2610.07774v1 Announce Type: new
Abstract: Vision-language models (VLMs) face compositional safety risks where harmful intent emerges from the interaction between visual and textual inputs. As m...
By Ziyuan Yang, Wenxuan Ding, Shangbin Feng, Yulia Tsvetkov
The paper introduces Counterfactual Neighborhood Risk Fields (CNRF), an evidence‑grounded framework that improves safety for vision‑language models (VLMs). CNRF builds context‑matched benign‑harmful counterfactual pairs, uses their representation differences to define risk directions, and separates neighborhood support from risk estimation. This approach allows selective intervention rather than blanket rejection, achieving high macro defense success rates across multiple VLMs while preserving benign‑task utility.
By Tianshi Wang, Jingsong Wang, Fengling Li, Jingjing Li, Xin Li, Lei Zhu
arXiv:2609.13231v1 Announce Type: cross
Abstract: Vision-Language-Action (VLA) models demonstrate strong generalization in robotic manipulation and navigation, but existing fine-tuning methods provid...
By Manan Tayal, Akshay Nambi
arXiv:2605.01913v2 Announce Type: replace-cross
Abstract: Fine-tuning safety-aligned language models for downstream tasks often leads to substantial degradation of refusal behavior, making models vul...
By Sadia Asif, Mohammad Mohammadi Amiri
arXiv:2609.00051v1 Announce Type: cross
Abstract: Despite extensive alignment efforts, Large Language Models (LLMs) remain vulnerable to generating unsafe content under adversarial prompting, yet the...
By Kuan-Lin Chu, Chung-En Sun, Tsui-Wei Weng
arXiv:2603. 18315v2 Announce Type: replace-cross Abstract: Traditional reinforcement learning (RL) methods rely on manually engineered rewards or sparse collision signals, which fail to capture the rich contextual understanding required for safe driving and make unsafe exploration unavoidable in real-world settings.
By Zilin Huang, Zihao Sheng, Zhengyang Wan, Yansong Qu, Junwei You, Sicong Jiang, Sikai Chen
As vision language models are increasingly deployed in clinical diagnosis, understanding how they internally resolve competing visual and textual signals becomes a safety imperative. Existing mechanis...