Omni‑SafetyBench is a new benchmark designed to evaluate the safety of Omni‑Modal Large Language Models (OLLMs) that process visual, auditory, and textual data. It contains 23,328 test instances across 24 modality variations derived from 972 seed samples, and introduces metrics such as Safety‑score (based on Conditional Attack Success Rate and Conditional Refusal Rate) and Cross‑Modal Safety Consistency score. Evaluation of 11 state‑of‑the‑art OLLMs shows severe vulnerabilities, with only three models achieving a Safety‑score above 0.6 and safety degrading sharply for audio‑visual inputs, underscoring the need for improved safety alignment methods.
By Leyi Pan, Zheyu Fu, Yunpeng Zhai, Shuchang Tao, Sheng Guan, Shiyu Huang, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Felix Henry, Aiwei Liu, Lijie Wen
arXiv:2608. 05381v1 Announce Type: new Abstract: Current Multimodal Large Language Models (MLLMs) can process diverse sensory inputs, yet their reasoning remains heavily biased toward a dominant modality, resulting in brittle cross-modal reasoning.
By Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu, Ponnurangam Kumaraguru
TRIAD is a new benchmark designed to test omni‑modal large language models’ ability to resolve ambiguity across text, vision, and audio. Each item consists of a text–image–audio triplet, a question, and answer options, constructed so that the full triplet yields a unique answer while any single modality removed makes the answer ambiguous. The benchmark includes 327 queries in 86 scene groups, covering 18 ambiguity categories, and shows that current models lag humans in scene‑level disambiguation, especially when answer options are removed.
By Zhaolu Kang, Yidi Wang, Yile Li, Siqi Zeng, Yingjie He, Richeng Xuan, Zhichao hu
arXiv:2609.36798v1 Announce Type: cross
Abstract: Omni-modal large language models (LLMs) are expected to answer a question using the modality it explicitly refers to. However, existing training para...
By Yueran Ma, Ronghao Lin
TRIAD is a new benchmark designed to test omni‑modal large language models’ ability to resolve ambiguity across text, vision, and audio. Each item consists of a text–image–audio triplet, a question, and answer options, crafted so that all three modalities together yield a unique answer while any single modality leaves the answer ambiguous. The benchmark contains 327 hand‑written queries in 86 scene groups, covering 18 ambiguity categories, and shows that current models lag humans significantly in scene‑level disambiguation, especially when answer options are removed.
arXiv:2610.31713v1 Announce Type: cross
Abstract: Advances in Omni-modal Large Language Models (OLLMs) have enabled increasingly seamless human-AI interaction, but their unified processing of heterog...
By Junhao Wang, Yidou Chen, Li Lu, Hongwei Yao, Xiaodan Li, Mengjie Wu, Yuefeng Chen, Zhan Qin, Kui Ren
arXiv:2606. 05177v1 Announce Type: cross Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text.
By Manh Luong, Tamas Abraham, Junae Kim, Amar Kaur, Rollin Omari, Gholamreza Haffari, Trang Vu, Lizhen Qu, Dinh Phung
arXiv:2609.06011v1 Announce Type: cross
Abstract: Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexp...
By Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen
The paper introduces Multi-Modal Typography, a systematic investigation into how perturbations applied to speech and on-screen text can compromise audio‑visual reasoning in large language models. Experiments on the WorldSense benchmark show that such attacks can raise the attack success rate against Gemini 3.7 Flash from 9.53% to 43.41%, and fine‑tuning can reduce the success rate for Qwen2.5‑Omni‑7B from 60.6% to 52.0%. The study highlights that cross‑modal typography attacks are a significant, yet underexplored, vulnerability in multi‑modal reasoning systems.
By Tianle Chen, Deepti Ghadiyaram
arXiv:2608.29286v1 Announce Type: new
Abstract: As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructi...
By Jinzhe Li, Gengxu Li, Jinnan Li, Yuan Wu, Yi Chang
The paper introduces IMAVB, a 500‑clip benchmark that tests whether omnimodal large language models can detect when a textual premise contradicts their visual or audio input. Experiments on eight open‑source models and Gemini 3.1 Pro reveal a Representation‑Action Gap: internal states encode mismatches, yet the models rarely reject false premises, exhibiting under‑rejection or over‑rejection. A probe‑guided logit adjustment improves rejection behavior, suggesting the main bottleneck is in translating perception to action rather than in perception itself.
By Trung Nguyen Quang, Yiming Gao, Fanyi Pu, Kaichen Zhang, Shuo Sun, Ziwei Liu
arXiv:2610.29249v1 Announce Type: new
Abstract: Omni-modal assistants must remember not only what users say, but also what they show and convey. Existing long-term memory benchmarks are mostly text-a...
By Hong Ting Tsang, Boyi Zhang, Haoyu Huang, Jiaxin Bai, Yufei Li, Yisen Gao, Zhongwei Xie, Hongyu Luo, Jeff Z. Pan, Yangqiu Song