← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,974 stories · RSS feed

arXiv Machine Learning

Sep 22

Beyond Raw Context Transfer: Representation-based Federated Retrieval-Augmented Generation

arXiv:2609.22162v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) improves the factuality of large language models (LLMs) and vision-language models (VLMs) by grounding generatio...

By Can Peng, Yu Liu, Yingyu Yang, Anjie Le, Yuyuan Liu, Qianye Yang, J. Alison Noble

llms rag nlp multimodal benchmarks

arXiv Machine Learning

Sep 22

Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models

arXiv:2609.22206v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable performance across a wide range of multimodal tasks, yet understanding and quantify...

By Soroush Seifi, Vaggelis Dorovatas, Lin Li, Yarin Gal, Rahaf Aljundi

llms multimodal benchmarks safety

arXiv Machine Learning

Sep 22

Assessing Adversarial Robustness of Latent Reasoning Models

arXiv:2609.22228v1 Announce Type: cross Abstract: Large language models increasingly rely on long chain-of-thought (CoT) trajectories for complex reasoning, but autoregressive generation brings subst...

By Shaolong Chen, Ang Li, Mingjie Li, Yisen Wang

llms multimodal benchmarks safety

arXiv Machine Learning

Sep 22

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

arXiv:2609.22234v1 Announce Type: cross Abstract: Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in...

By Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar

llms agents reinforcement-learning multimodal safety

arXiv Machine Learning

Sep 22

Scout: Open-World Species Recognition on the Edge

arXiv:2609.22897v1 Announce Type: cross Abstract: Large vision-language models (VLMs) enable recognition beyond a fixed class set, but their computational demands prevent them from running on many ed...

By Mohammad Mehdi Rastikerdar, Hui Guan, Deepak Ganesan

llms agents multimodal

arXiv Machine Learning

Sep 22

CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey

arXiv:2504.14280v2 Announce Type: replace-cross Abstract: As machine learning evolves, domain generalization (DG) and domain adaptation (DA) have become crucial for improving model robustness across...

By Jindong Li, Yongguang Li, Yali Fu, Jiahong Liu, Yixin Liu, Menglin Yang, Irwin King

llms fine-tuning multimodal safety

arXiv Computation and Language

Sep 22

COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning

arXiv:2609.22697v1 Announce Type: new Abstract: Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated...

By Weizhen Bian, Sitong Cheng, Rongxiu Zhong, Jiahao Pan, Liumeng Xue, Boyi Kang, Shilei Zhang, Jinglei Liu, Yue Wang, Junlan Feng, Bei Liu, Wei Xue

multimodal benchmarks

arXiv Computation and Language

Sep 22

Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba

arXiv:2609.23194v1 Announce Type: new Abstract: Acoustic representation learning is crucial for speech processing, yet low-resource languages (LRLs) face severe data scarcity, limiting the effectiven...

By Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie Tahon

multimodal benchmarks

arXiv Computation and Language

Sep 22

Long-Tail Rebalancing for Non-Verbal Vocalization-Aware ASR: A Track~1 System for the NVVSpeech Challenge

arXiv:2609.23462v1 Announce Type: cross Abstract: Non-verbal vocalizations (NVVs) carry important paralinguistic information but are often omitted by conventional automatic speech recognition (ASR) s...

By Shangyue Jia, Jingru Ma, Yangzhuo Li, Daoping Luo, Bowen Tian, Hanchen Lu, Wenze Ren, Yunxiang Chen, Houdun Liu, Shuo Feng, Lei Xie, Liumeng Xue

fine-tuning multimodal

arXiv Computation and Language

Sep 22

Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving

arXiv:2607.01733v2 Announce Type: replace Abstract: Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech reco...

By Ruchao Fan, Yiming Wang, Rui Zhao, Liliang Ren, Keqi Deng, Xiaoyang Chen, Ali Zare, Bo Ren, Yuxuan Hu, Junkun Chen, Yan Huang, Yelong Shen, Jinyu Li

llms nlp fine-tuning multimodal

arXiv Computation and Language

Sep 22

M-CIF: Multi-Scale Alignment For CIF-Based Non-Autoregressive ASR

arXiv:2510.22172v2 Announce Type: replace-cross Abstract: The Continuous Integrate-and-Fire (CIF) mechanism provides effective alignment for non-autoregressive (NAR) speech recognition. This mechanis...

By Ruixiang Mao, Xiangnan Ma, Qing Yang, Ziming Zhu, Yucheng Qiao, Yuan Ge, Tong Xiao, Shengxiang Gao, Zhengtao Yu, Jingbo Zhu

computer-vision multimodal safety

arXiv Computation and Language

Sep 22

X-GS: An Extensible Framework for Perceiving and Thinking with 3D Gaussian Splatting

arXiv:2603.09632v5 Announce Type: replace-cross Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, subsequently extending into numerous spatial AI ap...

By Yueen Ma, Zenglin Xu, Irwin King

efficiency multimodal benchmarks

arXiv Computer Vision

Sep 22

Vision-Wireless Fusion for Multi-User Localization: A Cross-Modal Transformer Approach

arXiv:2609.23372v1 Announce Type: new Abstract: Accurate multi-user localization is challenging in complex urban environments, where wireless measurements can become ambiguous under noise, blockage,...

By Can Zheng, Jiguang He, Guofa Cai, Henk Wymeersch, Merouane Debbah

llms multimodal

arXiv Computer Vision

Sep 22

ScaleBlind: Point Cloud Completion under Unknown Scale

arXiv:2609.23404v1 Announce Type: new Abstract: Point cloud completion aims to infer a complete 3D shape from a partial point cloud and serves as a fundamental building block for downstream tasks suc...

By Shenghui Wu, Chen Wang, Yuan Feng, Guangshun Wei, Yuanfeng Zhou, Changjian Li

diffusion multimodal benchmarks

arXiv Computer Vision

Sep 22

Detecting Phone-Induced Pedestrian Distraction via a Multimodal Fusion Transformer

arXiv:2609.23507v1 Announce Type: new Abstract: The increasing reliance on mobile phones has made phone-induced pedestrian distraction increasingly prevalent. Activities such as texting, watching vid...

By Yuanzhe Li, Hounian Liu, Xiaotong Chang, Yidi Huang

llms agents multimodal

arXiv Computer Vision

Sep 22

GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning

arXiv:2609.23533v1 Announce Type: new Abstract: Multimodal classifiers can converge to modality-dominant solutions in which one modality dominates the joint prediction, suppressing the learning of ot...

By Zechang Xiong, Da Li, Rong Yin, Kexin Tang, Biao Yang, Pengyuan Li, Wenkang Kong, Yulan Hu, Shengyu Zhu, Hao Peng

multimodal benchmarks

arXiv Computer Vision

Sep 22

SewFusion: Tailored Generation of Topology and Panel-Level Geometry for Sewing Patterns

arXiv:2609.23548v1 Announce Type: new Abstract: Generating sewing patterns from images and text requires modeling a heterogeneous representation composed of discrete topology and continuous geometry....

By Jiaxin Lin, Xiao Pan, Hangjie Yuan, Luyan Liang, Wan Li, Daquan Feng

diffusion multimodal benchmarks

arXiv Computer Vision

Sep 22

Which Terrain Is Better? Preference Learning with VLM Prototypes for Off-Road Traversability Ranking

arXiv:2609.23673v1 Announce Type: new Abstract: In vision-based off-road navigation, a robot needs to know not only which obstacles to avoid but also which terrain is better. The first is handled by...

By Ji-Hoon Hwang, Jisung Bae, E-In Son, Dong-Wook Kim, Jung-Taak Kim, Seung-Woo Seo

llms computer-vision robotics multimodal

arXiv Computer Vision

Sep 22

Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models

arXiv:2609.23715v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) incur substantial computational overhead due to the reliance on hundreds of visual tokens to represent images....

By Yahong Wang, Zhangkai Ni, Juncheng Wu, Yuyin Zhou, Ying Wen, Lianghua He

llms rag efficiency multimodal benchmarks

arXiv Computer Vision

Sep 22

DFD-Lab: A Modular Audio-Visual Deepfake Detection Pipeline

arXiv:2609.23830v1 Announce Type: new Abstract: Comparing audio-visual deepfake detectors requires coordinating dataset adaptation, temporal input representation, model interfaces and experimental co...

By Jan Rybarczyk, Mateusz Roszkowski, Jacek Komorowski