Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,891 stories · RSS feed

arXiv Machine Learning

Aug 4

SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

arXiv:2608. 00803v1 Announce Type: cross Abstract: Wearable silent speech interfaces (SSIs) are limited to small, closed vocabularies.

By Ruidong Zhang, Jiacheng Liu, Fran\c{c}ois Guimbreti\`ere, Cheng Zhang

multimodal benchmarks

arXiv Machine Learning

Aug 4

UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction

arXiv:2608. 01298v1 Announce Type: cross Abstract: Diffusion Transformers (DiTs) have emerged as a core architecture in generative modeling due to their scalability and adaptability to multimodal tasks.

By Junno Yun, Ya\c{s}ar Utku Al\c{c}alar, Mehmet Ak\c{c}akaya

llms diffusion multimodal safety

arXiv Machine Learning

Aug 4

Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI

arXiv:2509. 11924v2 Announce Type: cross Abstract: Multimodal learning has attracted much attention in recent years due to its ability to effectively utilize data features from a variety of different modalities.

By Bo Cao, Fan Yu, Mengmeng Feng, SenHao Zhang, Xin Meng, Yue Zhang, Zhen Qian, Jie Lu

efficiency multimodal

arXiv Machine Learning

Aug 4

Obshazard-bench: Benchmarking Multimodal Foundation Models for Real-Time Disaster Intelligence from Raw Earth Observation Streams

arXiv:2608. 00012v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are increasingly used to interpret Earth observation data, yet their capability to support real-world disaster emergency response remains insufficiently evaluated.

By Fengxiang Wang, Qiuyang Yu, Yueying Li, Mingshuo Chen, Chengchi Fei, Kaiyi Xu, Lixin Gu, Wangxu Wei, Junchao Gong, Lipeng Ma, Jiong Wang, Fenghua Ling, Wenlong Zhang, Xue Yang, Wenjing Yang, Ben Fei, Long Lan

llms multimodal benchmarks

arXiv Machine Learning

Aug 4

New York Smells: A Large Multimodal Dataset for Olfaction

arXiv:2511. 20544v2 Announce Type: replace-cross Abstract: While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines.

By Ege Ozguroglu, Junbang Liang, Ruoshi Liu, Mia Chiquier, Michael DeTienne, Wesley Wei Qian, Alexandra Horowitz, Andrew Owens, Carl Vondrick

multimodal benchmarks

arXiv Machine Learning

Aug 4

RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding

arXiv:2608. 00147v1 Announce Type: cross Abstract: Vision-language pretraining learns rich medical image representations from radiology reports, but previous model variants commonly operate within a single shared embedding space, so concept-level structure and interpretability must be recovered post hoc, limiting model transparency and, hence, clinical utility.

By Fabian Drexel, Marlene Fritzsche, Era Stambollxhiu, Miriam Kumpf, Lena Schmitzer, Lea Schumann, Jannik Kahmann, Friedrich Puttkammer, Johannes Moll, Jannik L\"ubberstedt, Zeineb Ben Chaaben, Anirudh Narayanan, Cosmin I. Bercea, Sebastian Ziegelmayer, Marcus R. Makowski, Daniel Rueckert, Lisa C. Adams, Keno K. Bressem

llms rag multimodal safety

arXiv Machine Learning

Aug 4

LOCUS-DT: Localization via Observation-Conditioned Uncertainty Scoring with Digital Twins

arXiv:2608. 00406v1 Announce Type: cross Abstract: Accurate indoor localization is essential for emerging applications in robotic navigation and search and rescue.

By Haozhe Lei, Roberto Bomfin, Marwa Chafii, Sundeep Rangan

robotics multimodal benchmarks

arXiv Machine Learning

Aug 4

RHEA: Reliability-Harmonized Reconstruction and Assignment for Robust Multimodal-Attributed Graph Clustering

arXiv:2608. 00621v1 Announce Type: new Abstract: Multimodal-attributed graphs (MAGs), whose nodes carry heterogeneous attributes such as text and images over a relational structure, have become a fundamental substrate for label-free entity grouping tasks, including community discovery and product segmentation.

By Yinlin Zhu, Di Wu, Ziyu Han, Zekai Chenm, Wang Luo, Miao Hu, Guocong Quan

computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records

arXiv:2506. 04831v3 Announce Type: replace Abstract: Forecasting how a patient's condition is likely to evolve, including possible deterioration, recovery, treatment needs, and care transitions, could support more proactive and personalized care, but requires modeling heterogeneous and longitudinal electronic health record (EHR) data.

By Chantal Pellegrini, Ege \"Ozsoy, David Bani-Harouni, Matthias Keicher, Nassir Navab

nlp multimodal

arXiv Machine Learning

Aug 4

Amortized Inference of Multi-Modal Posteriors using Likelihood-Weighted Normalizing Flows

arXiv:2512. 04954v3 Announce Type: replace Abstract: We present a novel technique for amortized posterior estimation using Normalizing Flows trained with likelihood-weighted importance sampling.

By Rajneil Baruah

multimodal benchmarks

arXiv Machine Learning

Aug 4

DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models

arXiv:2608. 01821v1 Announce Type: cross Abstract: Diffusion vision-language models (dVLMs) iteratively denoise masked responses while conditioning each denoising step on visual evidence, making visual conditioning a substantial recurring inference cost.

By Yongkang Zhou, Xiang Xia, Cheng Yan, Fan Xu, Wuyang Zhang

llms diffusion efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

Towards General Language-Conditioned Latent Safety Filters

arXiv:2608. 00315v1 Announce Type: cross Abstract: Robot policies are becoming increasingly general, with vision-language-action (VLA) models enabling a single policy to execute diverse tasks specified in natural language.

By Ihab Tabbara, Yuxuan Yang, Hussein Sibai

robotics multimodal safety

arXiv Machine Learning

Aug 4

Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study

arXiv:2608. 02235v1 Announce Type: cross Abstract: Recent advances in neural text-to-speech (TTS) systems have substantially improved speech naturalness and intelligibility across many languages.

By Ali Jafar, Amal Sarmad, Shifa Yousaf, Maryam Bashir

llms multimodal benchmarks

arXiv Machine Learning

Aug 4

D\'ej\`a Cue: Localizing States in Object Histories via Vocabulary-Relative Coordinates

arXiv:2608. 02044v1 Announce Type: cross Abstract: Tracking links observations of the same object through visual change, yet cannot by itself determine when the object is empty or filled, intact or cut.

By Haofan Cao, Zhichao You, Yunkai Yang, Liang Guo, Jie Wang, Chongshou Li

multimodal

arXiv Machine Learning

Aug 4

TELLER: Non-intrusive Cross-Layer Root-Cause Analysis for LLM Inference

arXiv:2608. 01975v1 Announce Type: cross Abstract: Large language model (LLM) inference has evolved from an offline workload into a continuously operated software service, yet root-cause analysis remains difficult because a single request spans the inference engine, Python/C++ backend, host CUDA APIs, GPU kernels, and distributed communication.

By Ruilin Xu, Junyi Li, Pengfei Chen, Zongxuan Xie

llms nlp multimodal

arXiv Machine Learning

Aug 4

ActionCache: Training-Free Acceleration for Vision-Language-Action Models with Action Caching and Refinement

arXiv:2607. 06370v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have emerged as a promising approach for generalizable robotic manipulations.

By Ryuji Oi, Hikari Otsuka, Kosuke Matsushima, Yuki Ichikawa, Masato Motomura, Tatsuya Kaneko, Daichi Fujiki

robotics efficiency multimodal

arXiv Machine Learning

Aug 4

Towards Effective Federated Multimodal Graph Learning via Navigating Multifaceted Heterogeneity

arXiv:2608. 00623v1 Announce Type: new Abstract: Multimodal-attributed graphs (MAGs), where nodes carry heterogeneous semantic content across multiple modalities while edges encode relational dependencies, have been widely adopted across diverse domains.

By Yinlin Zhu, Di Wu, Yi Zhang, Xunkai Li, Wang Luo, Wei-Jin Huang, Miao Hu, Guocong Quan

fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 4

nASR: An End-to-End Trainable Neural Layer for Channel-Level EEG Artifact Subspace Reconstruction in Real-Time BCI

arXiv:2605. 14941v2 Announce Type: replace-cross Abstract: Electroencephalogram (EEG) signals are highly susceptible to artifacts, resulting in a low signal-to-noise ratio, which makes extraction of meaningful neural information challenging.

By Shantanu Sarkar, Jose L. Contreras-Vidal

multimodal benchmarks

arXiv Machine Learning

Aug 4

GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs

arXiv:2608. 00877v1 Announce Type: new Abstract: Remote-sensing multimodal large language models (MLLMs) often assert facts that imagery cannot establish, such as a facility's identity or function.

By Xuechen Li

llms multimodal safety

arXiv Machine Learning

Aug 4

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

arXiv:2608. 01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories.

By Leyan Xue, Feng Xiong, Mingjun Ma, Changqing Zhang

llms efficiency multimodal benchmarks