arXiv Computation and Language August 25, 2026 By Leyi Pan, Zheyu Fu, Yunpeng Zhai, Shuchang Tao, Sheng Guan, Shiyu Huang, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Felix Henry, Aiwei Liu, Lijie Wen

Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models

Read the original on arXiv Computation and Language →

Omni‑SafetyBench is a new benchmark designed to evaluate the safety of Omni‑Modal Large Language Models (OLLMs) that process visual, auditory, and textual data. It contains 23,328 test instances across 24 modality variations derived from 972 seed samples, and introduces metrics such as Safety‑score (based on Conditional Attack Success Rate and Conditional Refusal Rate) and Cross‑Modal Safety Consistency score. Evaluation of 11 state‑of‑the‑art OLLMs shows severe vulnerabilities, with only three models achieving a Safety‑score above 0.6 and safety degrading sharply for audio‑visual inputs, underscoring the need for improved safety alignment methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jun 6

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

arXiv:2606. 05177v1 Announce Type: cross Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text.

By Manh Luong, Tamas Abraham, Junae Kim, Amar Kaur, Rollin Omari, Gholamreza Haffari, Trang Vu, Lizhen Qu, Dinh Phung

llms multimodal benchmarks safety

arXiv AI

3d ago

AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation

arXiv:2609.06991v1 Announce Type: cross Abstract: Recent text-to-audio-video (T2AV) models jointly generate video, speech, sound effects, and ambience from a single text prompt. This capability poses...

By Suah Choi, Tae-Young Lee, Gyeong-Moon Park

diffusion multimodal benchmarks safety

arXiv Computation and Language

Sep 1

Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning

arXiv:2608.29278v1 Announce Type: new Abstract: Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and read...

By Zhaolu Kang, Meixin Wu, Yu Xue, Yingjie He, Qiming Shi, Lei Wei, Yidi Wang, Richeng Xuan, Zhichao Hu

llms multimodal

arXiv AI

Aug 28

Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation

Multi2AV‑Safety is a new benchmark for evaluating safety in multimodal-to-audio‑video generation. It covers all 11 non‑singleton conditioning configurations (text, image, audio, video) and contains 11,024 attack instances. The benchmark reveals that safety guards often fail when harmful semantics arise from combinations of benign inputs or when explicit harmful cues are masked by benign multimodal context, highlighting a gap in compositional risk perception.

By Kaichao Jiang, Changtao Miao, Baiqi Wu, Zhiyuan Lu, Kang Yang, Peiwei Zhao, Junchi Chen, Yunfeng Diao, He Liu, Qi Chu, Tao Gong, Nenghai Yu

diffusion multimodal benchmarks safety

arXiv AI

Aug 10

CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment

arXiv:2604. 00310v2 Announce Type: replace-cross Abstract: Multimodal large-language models (MLLMs) often experience degraded safety alignment when harmful queries exploit cross-modal interactions.

By Anurag Kumar, Raghuveer Peri, Jon Burnsky, Alexandru Nelus, Rohit Paturi, Srikanth Vishnubhotla, Yanjun Qi

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Computation and Language

Sep 1

SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech

arXiv:2606.06037v3 Announce Type: replace-cross Abstract: Large audio language models (LALMs) are increasingly deployed in real-world applications, yet their safety alignment is still primarily evalu...

By Virginia Ceccatelli, Yejin Jeon, David Ifeoluwa Adelani

llms multimodal benchmarks safety

One email a morning, machine-written

Related stories

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation

Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning

Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation

CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment

SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech