arXiv:2608. 09227v1 Announce Type: new Abstract: Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive.
By Puneet Mathur, Manan Suri, Dinesh Manocha
arXiv:2607. 03050v1 Announce Type: cross Abstract: Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost.
By Shijie Cao, Qingyu Zhang, Boxi Yu, Yuzhong Zhang, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
arXiv:2609.37052v1 Announce Type: new
Abstract: Omnimodal large language models (Omni-LLMs) encode audio and visual streams into temporally interleaved token sequences for multimodal reasoning. Howev...
By Yuchen Deng, Zidang Cai, Feidiao Yang, Yufei Wang, Jie Wang, Hai-Tao Zheng, Yuxing Han
arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.
By Kaiying Yan, Luoyi Sun, Xiao Zhou, Weidi Xie
The paper introduces Omni-Interactive Universal Embedder (OmniUE), a unified embedding framework that learns a single representation space for text, video, and audio using learnable tokens and intermediate-layer representations. OmniUE supports omni-interactive querying, allowing users to input text, visual regions, or audio spans, which are processed by segmenters and an omni-LLM to generate user-conditioned embeddings. The authors evaluate OmniUE on the new OmniCHOIR benchmark and other multimodal tasks, reporting significant performance gains over state‑of‑the‑art baselines across textual, audio, and visual interactive settings.
By Wei-Yao Wang, Kazuya Tateishi, Shuyang Cui, Christian Simon, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji
The paper introduces Training-Free Omni (TFO), a plug‑and‑play framework that transforms a frozen vision‑language model (VLM) into a speech‑centric omni model without modifying its architecture or requiring multimodal re‑alignment. TFO leverages Whisper to generate confidence‑filtered, timestamped transcripts and routes them through the VLM’s existing language interface, leaving the visual pathway untouched. Evaluations on 56 benchmarks across 21 languages show that TFO matches or surpasses native omni models on audio‑visual tasks, improves audio‑only performance, and preserves strong visual and reasoning capabilities.
By Ankan Deria, Hanoona Rasheed, Xilin He, Fahad Shahbaz Khan, Salman Khan