Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv AI

Sep 10

AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation

arXiv:2609.06991v1 Announce Type: cross Abstract: Recent text-to-audio-video (T2AV) models jointly generate video, speech, sound effects, and ambience from a single text prompt. This capability poses...

By Suah Choi, Tae-Young Lee, Gyeong-Moon Park

diffusion multimodal benchmarks safety

arXiv AI

Sep 10

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

arXiv:2609.07047v1 Announce Type: cross Abstract: Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when t...

By Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, Zhou Zhao

robotics multimodal benchmarks safety

arXiv AI

Sep 10

Federated Binary Gating with Server-Side Vision-Language Inference for Surveillance Anomaly Classification

arXiv:2609.07403v1 Announce Type: cross Abstract: Privacy-sensitive surveillance systems could benefit from large vision-language models (VLMs), but such models typically require centralized access t...

By C\^ome-Alexis Puech, S\'ebastien Thuau, Amira Gran, Arthur Mennessier, Siba Haidar, Rachid Chelouah

llms multimodal safety

arXiv AI

Sep 10

Zero-Shot Sim-to-Real Contact-Rich Assembly via Proprioception-Anchored Cross-Modal Pretraining

arXiv:2609.07534v1 Announce Type: cross Abstract: Contact-rich assembly remains challenging because it requires submillimeter spatial accuracy and reliable interpretation of forces during sustained c...

By Yuhan Wang, Yurou Chen, Hongye Jiang, Wenzhao Lian

reinforcement-learning robotics fine-tuning multimodal safety

arXiv AI

Sep 10

DISEIL: Demonstration Distillation for Sample-Efficient Imitation Learning

arXiv:2609.08123v1 Announce Type: cross Abstract: A robot that can be taught a new task from a handful of demonstrations has to work out for itself what it still cannot do, and then ask for exactly t...

By Suyog Khanal, Arun Kumar A V, Santu Rana

llms robotics efficiency multimodal

arXiv AI

Sep 10

Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method

arXiv:2609.08402v1 Announce Type: cross Abstract: Air-Ground Object Search (AGOS) in urban environments is a challenging embodied task, which requires an Unmanned Aerial Vehicle (UAV) and an Unmanned...

By Boao Yu, Zimo Chen, Junreng Rao, Yue Hu, Zhengqiu Zhu, Yong Zhao, Rusheng Ju

llms agents robotics multimodal benchmarks

arXiv AI

Sep 10

DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination

arXiv:2609.09119v1 Announce Type: cross Abstract: Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision...

By Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang

llms robotics multimodal benchmarks

arXiv AI

Sep 10

Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

arXiv:2609.09124v1 Announce Type: cross Abstract: Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, inf...

By Xiaofu Chen, Stella Frank, Yova Kementchedjhieva

llms multimodal

arXiv AI

Sep 10

Empowering VLMs for Few-Shot Multimodal Time Series Classification via Tailored Agentic Reasoning

arXiv:2605.09395v3 Announce Type: replace Abstract: In this paper, we propose the first VL\underline{\textbf{M}} \underline{\textbf{a}}gentic \underline{\textbf{r}}easoning framework for few-\underli...

By Lin Li, Jiawei Huang, Qihao Quan, Dan Li, Boxin Li, Xiao Zhang, Erli Meng, Wenjie Feng, Jian Lou, See-Kiong Ng

agents multimodal benchmarks safety

arXiv AI

Sep 10

Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning

arXiv:2510.11454v2 Announce Type: replace-cross Abstract: Recent advancements in large multimodal models (LMMs) have shown strong capabilities in audio understanding. However, most systems rely solel...

By Kuan-Yi Lee, Tsung-En Lin, Hung-Yi Lee

llms multimodal safety

arXiv AI

Sep 10

MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data

arXiv:2601.06847v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) can generate convincing clinical narratives, yet frequently struggle to visually ground their statements. We po...

By Mengmeng Zhang, Xiaoping Wu, Hao Luo, Fan Wang, Yisheng Lv

llms computer-vision multimodal

arXiv Machine Learning

Sep 10

CAVEAT: Recurrent Multimodal Diffusion Planning for Mapless Aerial Exploration

arXiv:2609.06623v1 Announce Type: cross Abstract: Can exploratory UAV waypoint sequences be generated from multimodal onboard observations and a fixed-dimensional recurrent internal state without mai...

By Steven Visch, Nicol\`o Botteghi, Antonio Franchi, Barbara Bazzana

diffusion multimodal

arXiv AI

Sep 10

Spatiotemporal Heterogeneity of AI-Driven Traffic Flow Patterns and Land Use Interaction: A GeoAI-Based Analysis of Multimodal Urban Mobility

arXiv:2603.05581v2 Announce Type: replace-cross Abstract: Urban traffic flow is governed by the complex, nonlinear interaction between land use configuration and spatiotemporally heterogeneous mobili...

By Olaf Yunus Laitinen Imanov

multimodal benchmarks

arXiv AI

Sep 10

Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

arXiv:2605.27750v2 Announce Type: replace-cross Abstract: Recent work has shown that Vision-Language Models (VLMs) used for optical character recognition (OCR) can generate plausible but visually uns...

By Antonia Karamolegkou, Nicolas Angleraud, Beno\^it Sagot, Thibault Cl\'erice

llms computer-vision multimodal safety

arXiv Machine Learning

Sep 10

NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts

arXiv:2609.07009v1 Announce Type: new Abstract: Multimodal continual learning has recently shown great potential for developing agents with human-like intelligence by continuously learning new tasks...

By Kai Guo, Chuanbin Liu, Peng Hu, Hao Wang, Xi Peng

agents efficiency multimodal benchmarks

arXiv Machine Learning

Sep 10

InfluenceField: A Differentiable Field with Interventionally Identifiable Causal Structure for Multimodal World Modeling

arXiv:2609.07874v1 Announce Type: new Abstract: Multimodal large language models often capture visual-linguistic correlations but struggle to predict how local visual interventions propagate and affe...

By Zihao Yang, Zijia Wang, Zhiqiu Huang

llms multimodal

arXiv Machine Learning

Sep 10

GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them

arXiv:2609.06058v1 Announce Type: cross Abstract: As vision-language models (VLMs) become increasingly capable and are deployed in consequential real-world settings, they must evaluate evidence indep...

By Deep Dessai (The University of Texas at Austin)

llms multimodal

arXiv Machine Learning

Sep 10

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

arXiv:2609.06880v1 Announce Type: cross Abstract: Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated pers...

By Mimo Shirasaka, Haochen Zhang, Yonatan Bisk

llms robotics multimodal benchmarks

arXiv Computation and Language

Sep 10

StreamAlign: Streaming Text-Aligned Speech Tokenization

arXiv:2609.09719v1 Announce Type: new Abstract: Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretr...

By Kang-wook Kim, Jinyoung Park, Jinsoo Kim, Sehun Lee, Sang Hoon Woo, Gunhee Kim

llms nlp multimodal safety

arXiv Machine Learning

Sep 10

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

arXiv:2509.10452v3 Announce Type: replace-cross Abstract: Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance....

By Akshat Pandey, Karun Kumar, Raphael Tang

llms fine-tuning multimodal

Get the daily briefing