arXiv:2606. 05177v1 Announce Type: cross Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
Omni‑SafetyBench is a new benchmark designed to evaluate the safety of Omni‑Modal Large Language Models (OLLMs) that process visual, auditory, and textual data. It contains 23,328 test instances across 24 modality variations derived from 972 seed samples, and introduces metrics such as Safety‑score (based on Conditional Attack Success Rate and Conditional Refusal Rate) and Cross‑Modal Safety Consistency score. Evaluation of 11 state‑of‑the‑art OLLMs shows severe vulnerabilities, with only three models achieving a Safety‑score above 0.6 and safety degrading sharply for audio‑visual inputs, underscoring the need for improved safety alignment methods.
The paper introduces Uni-SafeBench, a safety benchmark designed to evaluate the holistic safety of Unified Multimodal Large Models (UMLMs) across six safety categories and seven task types. It also presents Uni-Judger, a framework that separates contextual safety from intrinsic safety to enable rigorous assessment. Evaluations reveal that current unified models do not consistently maintain the safety alignment of their underlying language models, and open‑source UMLMs perform significantly worse on safety than specialized multimodal models, especially in generation tasks.
arXiv:2609.20850v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easi...
The paper investigates cross‑modal safety drift in multimodal large language models, where a harmless text query paired with a visual image can trigger harmful responses. Empirical analysis identifies unsafe response patterns and shows that visual cues receive limited attention, weakening refusal mechanisms. The authors introduce Safety‑Awareness Representation Transfer (SRT), a lightweight method that transfers safety signals from text processing to mitigate cross‑modal drift while maintaining model utility.
SafeAtlas-VL introduces a large multimodal safety dataset with 1.5 million instances, rating image, request, and response risks on a five‑level ordinal scale across 15 harm categories and 55 subcategories. The accompanying SafeAtlas‑Bench provides 5,000 held‑out cases for evaluating ordinal predictions and continuous risk scores. Models trained on this data, including an 8B Guard model, achieve state‑of‑the‑art performance, outperforming prior benchmarks by about 4% in F1 score.
The paper introduces Safety-aware Contrastive Decoding (SafeCoDe), a lightweight, model‑agnostic framework designed to improve context‑aware safety in Multimodal Large Language Models (MLLMs). SafeCoDe operates in two stages: a contrastive decoding step that highlights tokens sensitive to visual context by contrasting real and Gaussian‑noised images, and a global‑aware token modulation strategy that adjusts refusals based on scene‑level reasoning and predicted safety verdicts. Experiments across various MLLM architectures and safety benchmarks demonstrate that SafeCoDe consistently enhances context‑sensitive refusal behaviors while maintaining model helpfulness.