The paper investigates cross‑modal safety drift in multimodal large language models, where a harmless text query paired with a visual image can trigger harmful responses. Empirical analysis identifies unsafe response patterns and shows that visual cues receive limited attention, weakening refusal mechanisms. The authors introduce Safety‑Awareness Representation Transfer (SRT), a lightweight method that transfers safety signals from text processing to mitigate cross‑modal drift while maintaining model utility.
By Tianqi Xiao, Shiyao Cui, Minghao Zhang, Junxiao Yang, Renmiao Chen
The paper introduces SAFE-PRO, a method for enhancing safety in multimodal large language models by proactively asking clarifying questions when cross‑modal ambiguity poses intent‑level risks. SAFE‑PRO employs a Risk‑Conditioned Uncertainty signal that blends predictive entropy with a learned cross‑modal risk score to guide both question‑asking and path optimization, prioritizing safety while reducing over‑refusal. The authors also present the SSUI‑Dialog dataset and demonstrate that SAFE‑PRO outperforms existing approaches, including GPT‑4o and Claude‑3.5‑Sonnet, across multiple benchmarks.
By Wei Cai, Jian Zhao, Yuchen Yuan, Xuelong Li
The paper introduces MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images across 12 high-risk domains, each paired with a hidden user profile, to evaluate personalized safety in vision‑language models (VLMs). Eight leading VLMs were tested and found to almost always respond directly (86‑99%) without seeking missing context, scoring no higher than 2.6/5 on personalized safety. The authors identify a phenomenon called visual dominance, where visual information enters text representations early and suppresses textual risk signals, and propose PRISM, a lightweight input monitor that predicts when a query should be deferred, achieving 0.978 AUC and outperforming all tested models on the safety‑utility Pareto frontier.
By Edward Sun, Yuchen Wu, Zixian Ma, Eric Hanchen Jiang, Yijia Xiao, Xiaoyuan Yi, Ranjay Krishna, Wei Wang, Jindong Wang, Aylin Caliskan
arXiv:2606. 31876v1 Announce Type: new Abstract: To improve safety in Large Language Models (LLMs) we can either perform post-training alignment or exploit refusal directions in the activation space.
By Moreno D'Inc\`a, Massimiliano Mancini, Nicu Sebe
Multimodal large language models (MLLMs) can act unsafely when individually benign images and texts are interpreted jointly. Beyond implicit reasoning risk, where reasoning drifts toward harm, we iden...
ReFrame is a training‑free framework that enhances safety alignment for multimodal large language models at test time. It uses two lightweight agents: one generates risk and utility evidence, and the other rewrites prompts and routes images to create a safe proxy before invoking the deployed MLLM. Experiments show that ReFrame improves jailbreak defense, safety awareness, and reduces over‑sensitivity while maintaining multimodal utility.
By Wenzheng Jiang, Xuankun Rong, Yuanzhao Zhai, Dawei Feng, Huaimin Wang