Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,869 stories · RSS feed

arXiv Machine Learning

Aug 4

GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding

arXiv:2605. 13352v2 Announce Type: replace Abstract: Standard dual-encoder vision-language models that map images and text to deterministic points on a shared unit hypersphere through $\ell_2$ normalization typically expose neither \emph{aleatoric} uncertainty (cross-modal ambiguity) nor \emph{epistemic} uncertainty (lack of training-distribution support).

By Mayank Nautiyal, Li Ju, Andreas Hellander, Ekta Vats, Prashant Singh

llms rag diffusion multimodal benchmarks

arXiv Machine Learning

Aug 4

RHEA: Reliability-Harmonized Reconstruction and Assignment for Robust Multimodal-Attributed Graph Clustering

arXiv:2608. 00621v1 Announce Type: new Abstract: Multimodal-attributed graphs (MAGs), whose nodes carry heterogeneous attributes such as text and images over a relational structure, have become a fundamental substrate for label-free entity grouping tasks, including community discovery and product segmentation.

By Yinlin Zhu, Di Wu, Ziyu Han, Zekai Chenm, Wang Luo, Miao Hu, Guocong Quan

computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Aug 4

ActionCache: Training-Free Acceleration for Vision-Language-Action Models with Action Caching and Refinement

arXiv:2607. 06370v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have emerged as a promising approach for generalizable robotic manipulations.

By Ryuji Oi, Hikari Otsuka, Kosuke Matsushima, Yuki Ichikawa, Masato Motomura, Tatsuya Kaneko, Daichi Fujiki

robotics efficiency multimodal

arXiv Machine Learning

Aug 4

Amortized Inference of Multi-Modal Posteriors using Likelihood-Weighted Normalizing Flows

arXiv:2512. 04954v3 Announce Type: replace Abstract: We present a novel technique for amortized posterior estimation using Normalizing Flows trained with likelihood-weighted importance sampling.

By Rajneil Baruah

multimodal benchmarks

arXiv Machine Learning

Aug 4

SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

arXiv:2608. 00803v1 Announce Type: cross Abstract: Wearable silent speech interfaces (SSIs) are limited to small, closed vocabularies.

By Ruidong Zhang, Jiacheng Liu, Fran\c{c}ois Guimbreti\`ere, Cheng Zhang

multimodal benchmarks

arXiv Machine Learning

Aug 4

Hierarchical Pre-Training of Vision Encoders with Large Language Model

arXiv:2604. 00086v2 Announce Type: replace-cross Abstract: The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks.

By Eugene Lee, Ting-Yu Chang, Jui-Huang Tsai, Jiajie Diao, Chen-Yi Lee

llms rag computer-vision multimodal benchmarks safety

arXiv Machine Learning

Aug 4

GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs

arXiv:2608. 00877v1 Announce Type: new Abstract: Remote-sensing multimodal large language models (MLLMs) often assert facts that imagery cannot establish, such as a facility's identity or function.

By Xuechen Li

llms multimodal safety

arXiv Machine Learning

Aug 4

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

arXiv:2608. 01899v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning.

By Jing Wu, Jianhua Wu, Jiayi Guan, Jiahong Chen, Jinghui Lu, Hangjun Ye, Bingzhao Gao, Long Chen

llms robotics fine-tuning multimodal

arXiv Machine Learning

Aug 4

Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

arXiv:2608. 01585v1 Announce Type: cross Abstract: Language model benchmarking is a difficult task.

By Tyler Ashoff, Jordan Rodu

llms rag multimodal benchmarks safety

arXiv Machine Learning

Aug 4

Towards General Language-Conditioned Latent Safety Filters

arXiv:2608. 00315v1 Announce Type: cross Abstract: Robot policies are becoming increasingly general, with vision-language-action (VLA) models enabling a single policy to execute diverse tasks specified in natural language.

By Ihab Tabbara, Yuxuan Yang, Hussein Sibai

robotics multimodal safety

arXiv Machine Learning

Aug 4

AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference

arXiv:2604. 15622v3 Announce Type: replace-cross Abstract: Always-on contextual AI runs language-aligned vision foundation models (VFMs) on edge devices, where the on-device model is the dominant continuous compute cost under strict latency and power limits.

By Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michael Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li

llms computer-vision multimodal

arXiv Machine Learning

Aug 4

LOCUS-DT: Localization via Observation-Conditioned Uncertainty Scoring with Digital Twins

arXiv:2608. 00406v1 Announce Type: cross Abstract: Accurate indoor localization is essential for emerging applications in robotic navigation and search and rescue.

By Haozhe Lei, Roberto Bomfin, Marwa Chafii, Sundeep Rangan

robotics multimodal benchmarks

arXiv Machine Learning

Aug 4

Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages

arXiv:2608. 01281v1 Announce Type: cross Abstract: Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling.

By Saierdaer Yusuyin, Nanling Jiang, Hao Huang, Zhijian Ou

multimodal

arXiv Machine Learning

Aug 4

Obshazard-bench: Benchmarking Multimodal Foundation Models for Real-Time Disaster Intelligence from Raw Earth Observation Streams

arXiv:2608. 00012v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are increasingly used to interpret Earth observation data, yet their capability to support real-world disaster emergency response remains insufficiently evaluated.

By Fengxiang Wang, Qiuyang Yu, Yueying Li, Mingshuo Chen, Chengchi Fei, Kaiyi Xu, Lixin Gu, Wangxu Wei, Junchao Gong, Lipeng Ma, Jiong Wang, Fenghua Ling, Wenlong Zhang, Xue Yang, Wenjing Yang, Ben Fei, Long Lan

llms multimodal benchmarks

arXiv Machine Learning

Aug 4

ARMOR: A Robust Self-Supervised Framework for Root Cause Analysis in Microservices under Missing Modality

arXiv:2603. 25538v3 Announce Type: replace Abstract: Automated incident management is critical for microservice reliability.

By Wenzhuo Qian, Hailiang Zhao, Ziqi Wang, Zhipeng Gao, Jiayi Chen, Zhiwei Ling, Shuiguang Deng

agents multimodal benchmarks safety

arXiv Machine Learning

Aug 4

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

arXiv:2606. 21337v2 Announce Type: replace Abstract: Raw multimodal streams are abundant but noisy, redundant, and unaligned with any particular training objective.

By Cong Wan, Zeyu Guo, Zijian Cai, Jiangyang Li, SongLin Dong, Lin Peng, Xiangyang Luo, Zhiheng Ma, Yihong Gong

llms agents diffusion multimodal

arXiv Machine Learning

Aug 4

Element-Aware Group Learning for E-Commerce Image Generation

arXiv:2608. 00584v1 Announce Type: cross Abstract: Recent advances in image generation and editing have made prompt quality a key bottleneck for e-commerce creatives.

By Jingtong Chen, Jiahui Wang, Xue Zhao, ShaoGuo Liu, Minghao Li

llms diffusion multimodal

arXiv Machine Learning

Aug 4

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

arXiv:2603. 11804v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce.

By Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Delyan Boychev (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Aug 4

nASR: An End-to-End Trainable Neural Layer for Channel-Level EEG Artifact Subspace Reconstruction in Real-Time BCI

arXiv:2605. 14941v2 Announce Type: replace-cross Abstract: Electroencephalogram (EEG) signals are highly susceptible to artifacts, resulting in a low signal-to-noise ratio, which makes extraction of meaningful neural information challenging.

By Shantanu Sarkar, Jose L. Contreras-Vidal

multimodal benchmarks

arXiv Machine Learning

Aug 4

FAU at ImageCLEF 2026 Task on Multimodal Reasoning Robust Candidate Scoring and Concise Multilingual Visual Answering

arXiv:2608. 01664v1 Announce Type: cross Abstract: We present our ImageCLEF 2026 Multimodal Reasoning system for the Visual Multiple Choice Question Answering (Visual MCQ) and Visual Open Question Answering (Visual OpenQA) subtasks.

By Mohamed Basem, Vincent Christlein

llms nlp multimodal

Get the daily briefing