Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,974 stories · RSS feed

arXiv Computation and Language

Sep 22

The Bairong System for MLC-SLM 2026: Dynamic Question-Aware Evidence Routing for Multilingual Conversational Speech Understanding

arXiv:2609.22214v1 Announce Type: new Abstract: Long multilingual conversational spoken question answering requires systems to balance long-range transcript semantics with sparse acoustic and speaker...

By Shangkun Huang, Junchao Hu, Huan Shen, Guoji Wang, Yingao Wang, Shaosai Li, Wei Zou, Yunzhang Chen

nlp multimodal

arXiv Computation and Language

Sep 22

MIS-Bench: Benchmarking Multimodal LLMs for Psychotherapeutic Interpersonal Skills Assessment

arXiv:2609.22778v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require exper...

By Yuhan Lu, Yi Yao, Hua Shen, Katie Aafjes-van Doorn, Zhaonan Wang

llms fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 22

Automatic multimodal UX improvement recommendations from LLM agent user simulations

arXiv:2609.22971v1 Announce Type: new Abstract: Evaluating user experience (UX) on live websites through user testing is expensive, subjective, and difficult to scale. LLM agents offer a promising ro...

By Anu Chowdhury, Bin Wu, Hossein A. Rahmani, Emine Yilmaz

llms agents multimodal

arXiv Computation and Language

Sep 22

Low resource cross-modal alignment using HGNN to enhance speech representation

arXiv:2609.23191v1 Announce Type: new Abstract: Speech-text space alignment is a multimodal representation learning method consisting to map different speech and text into a shared representation spa...

By Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia Yonta

fine-tuning multimodal benchmarks safety

arXiv Computation and Language

Sep 22

Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment

arXiv:2609.23267v1 Announce Type: new Abstract: The visual modality, i.e., images, plays a key role in multi-modal entity alignment (MMEA). Existing approaches often directly fuse the image with othe...

By Chenxiao Li, Yunhe Feng, Dongfang Liu, Dong Nie, Yan Huang, Heng Fan

rag multimodal benchmarks safety

arXiv Computation and Language

Sep 22

HaikuS2S: A Cascaded System For Responding In Verse

arXiv:2609.23951v1 Announce Type: new Abstract: Expressive speech synthesis has advanced through prosody modeling, yet generating structured poetic speech, such as haiku, remains challenging. Prior w...

By Devangi Sharma, Sophia Judicke, Glenda Tan, Conrad Schaumburg, Shinji Watanabe

llms fine-tuning multimodal safety

arXiv Computation and Language

Sep 22

Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

arXiv:2609.24199v1 Announce Type: new Abstract: Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlle...

By Kaushal Santosh Bhogale, Srija Anand, Sadakopa Ramakrishnan Thothathiri, Tahir Javed, Sshubam Verma, Mitesh M. Khapra

multimodal benchmarks

arXiv Computation and Language

Sep 22

Structure Before Sampling: Community-Aware Core-Set Selection for Data-Efficient Text-to-Speech

arXiv:2609.24275v1 Announce Type: new Abstract: Text-to-speech (TTS) corpora are costly to record, yet many utterances add little new phonetic information. Core-set selection reduces this cost by cho...

By Mizbaul Haque Maruf, Muhammad Nur Yanhaona

multimodal

arXiv Computation and Language

Sep 22

AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Models

arXiv:2609.23979v1 Announce Type: cross Abstract: Attention encoder-decoder (AED) Speech Foundation Models achieve strong ASR performance but can generate acoustically unsupported text when inputs co...

By Natarajan Balaji Shankar, Zilai Wang, Zihan Wang, Mohan Shi, Kaiyuan Zhang, Abeer Alwan

fine-tuning multimodal safety

arXiv Computation and Language

Sep 22

Morpho-VITS: Variational Inference with Morphological Modeling for End-to-End Speech Synthesis of a Tonal Bantu Language

arXiv:2609.24310v1 Announce Type: cross Abstract: Text-to-speech models for Bantu tonal languages are challenged by a tonal system that is rooted in both the lexis (i.e., the inventory of words, stem...

By Antoine Nzeyimana

multimodal

arXiv Computation and Language

Sep 22

SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models

arXiv:2609.24894v1 Announce Type: cross Abstract: Whole-slide pathology images (WSIs) contain gigapixel-scale visual content, creating a major scalability challenge for slide-level multimodal large l...

By Ali Kerem Bozkurt, Baris Cem Bakay, Ibrahim Kulac, Cigdem Gunduz-Demir, Erkut Erdem, Aykut Erdem

llms efficiency multimodal

arXiv Computation and Language

Sep 22

3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation

arXiv:2501.16698v2 Announce Type: replace Abstract: Spatial intelligence, encompassing 3D perception and reasoning, is the essential next frontier of AI. Scaling current 3D vision-language models (VL...

By Yueen Ma, Zenglin Xu, Irwin King

llms diffusion robotics multimodal benchmarks

arXiv Computation and Language

Sep 22

MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine

arXiv:2603.00842v2 Announce Type: replace Abstract: Biomedical multimodal assistants have the potential to unify radiology, pathology, and clinical-text reasoning, yet a critical deployment gap remai...

By Kai Zhang, Zhengqing Yuan, Cheng Peng, Songlin Zhao, Mengxian Lyu, Ziyi Chen, Yanfang Ye, Wei Liu, Ying Zhang, Kaleb E Smith, Lifang He, Lichao Sun, Yonghui Wu

llms multimodal safety

arXiv Computation and Language

Sep 22

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

arXiv:2604.06487v2 Announce Type: replace Abstract: Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR arch...

By Thibault Ba\~neras-Roux, Sergio Burdisso, Esa\'u Villatoro-Tello, Dairazalia S\'anchez-Cort\'es, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke

llms fine-tuning multimodal safety

arXiv Computation and Language

Sep 22

Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

arXiv:2605.10893v3 Announce Type: replace Abstract: Large vision-language models (LVLMs) suffer from visual ungroundedness: they can produce a fluent, confident, and even correct response driven enti...

By Reza Khanmohammadi, Erfan Miahi, Simerjot Kaur, Charese H. Smiley, Ivan Brugere, Kundan Thind, Mohammad M. Ghassemi

llms nlp multimodal benchmarks safety

arXiv Computation and Language

Sep 22

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

arXiv:2508.03351v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-la...

By Yufei Xue, Yushi Huang, Lunjie Zhu, Jiawei Shao, Jun Zhang

llms efficiency multimodal benchmarks

arXiv Computer Vision

Sep 22

Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models

arXiv:2609.22293v1 Announce Type: new Abstract: Vision-language models (VLMs) and vision-language-action models (VLAs) are increasingly deployed in real-world applications. There, a small perturbatio...

By Bogdan Aron, Christopher Brix, Benedikt Br\"uckner, Yanghao Zhang, Panagiotis Kouvaros, Alessio Lomuscio

llms multimodal benchmarks

arXiv Computer Vision

Sep 22

Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphs

arXiv:2609.22351v1 Announce Type: new Abstract: Open-vocabulary 3D Scene Graphs (3DSGs) ground each object node in a vision-language embedding, yet they record every entry as equally certain, so a ro...

By Carlos Cueto Zumaya, Iacopo Catalano, Wallace Moreira Bessa, Julio A. Placed

llms rag robotics multimodal benchmarks

arXiv Computer Vision

Sep 22

GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents

arXiv:2609.22308v1 Announce Type: new Abstract: Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing researc...

By Boyu Qiao, Zixin Tang, Xiaoshuai Hao, Wenbo Li

llms agents multimodal benchmarks

arXiv Computer Vision

Sep 22

Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act

arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...

By Yuyang Dai, Bofei Huang, Hongbo Zhang, Haoran Xie

llms nlp multimodal benchmarks

Get the daily briefing