arXiv AI October 9, 2026 By Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji

Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

Syn-Omni introduces a structured approach to omnimodal embeddings by separating universal semantics from modality-specific features through Orthogonal Modality-Expert LoRA (OME-LoRA). It further employs Progressive Synergy Routing (PSR) to allow modality experts to first learn modality-specific priors and then gradually collaborate across modalities. Across 81 tasks involving image, video, audio, and audiovisual data, Syn-Omni outperforms existing omnimodal baselines, highlighting the benefits of structured specialization and progressive cross-modal collaboration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 28

Omni-Interactive Universal Embedder

The paper introduces Omni-Interactive Universal Embedder (OmniUE), a unified embedding framework that learns a single representation space for text, video, and audio using learnable tokens and intermediate-layer representations. OmniUE supports omni-interactive querying, allowing users to input text, visual regions, or audio spans, which are processed by segmenters and an omni-LLM to generate user-conditioned embeddings. The authors evaluate OmniUE on the new OmniCHOIR benchmark and other multimodal tasks, reporting significant performance gains over state‑of‑the‑art baselines across textual, audio, and visual interactive settings.

By Wei-Yao Wang, Kazuya Tateishi, Shuyang Cui, Christian Simon, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji

llms rag multimodal benchmarks

arXiv AI

Jun 2

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.

By Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

llms multimodal benchmarks safety

arXiv Computer Vision

Oct 2

Gestalt: Large Multimodal Interplay Model

arXiv:2610.00576v1 Announce Type: new Abstract: In this paper, we propose Gestalt, a new paradigm of large multimodal model built around multimodal interplay. Despite rapid advances, large multimodal...

By Zequn Yang, Yu Miao, Haotian Ni, Ziheng Chen, Chengxiang Huang, Dongzhan Zhou, Kai Chen, Qi Zhang, Ji-Rong Wen, Yake Wei, Di Hu

diffusion multimodal safety

arXiv Computer Vision

Sep 17

Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

arXiv:2609.18323v1 Announce Type: new Abstract: Recent Omni-Modal Generative Models (Omni-Models) have advanced content generation toward unified modeling of text, images, video, and audio. MiniMax-H...

By Haoyu Zhao, Zihao Zhao, Tianyu Deng, Ziqin Xu, Zihao Zhang, Xudong Wang, Jinxiang Guo, Chen Gao, Ziyi Ye, Yeying Jin, Jiaxi Gu, Zuxuan Wu, Shuicheng Yan

diffusion multimodal safety

arXiv Machine Learning

Jul 14

Empowering Long-form Omni-modal Understanding with Robust Audio Perception

arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.

By Kaiying Yan, Luoyi Sun, Xiao Zhou, Weidi Xie

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 2

Rosetta: Composable Native Multimodal Pretraining

arXiv:2607. 00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge.

By Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Ping Tan

llms diffusion multimodal