Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,393 stories · RSS feed

arXiv AI

Jun 11

Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

arXiv:2606. 12412v1 Announce Type: cross Abstract: Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory.

By Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

llms efficiency multimodal

arXiv AI

Jun 11

AutoMine Solution for AV2 2026 Scenario Mining Challenge

arXiv:2606. 11874v1 Announce Type: new Abstract: With the development of autonomous driving systems, mining high-value, safety-critical, and planning-relevant scenarios from large-scale driving logs has become essential for data-driven evaluation.

By Songliang Cao, Jiele Zhao, Yuru Wang, Hao Li, Daqi Liu, Zehan Zhang, Fangzhen Li, Yu Wang, Yue Zhang, Bing Wang, Guang Chen, Hao Lu, Hangjun Ye

llms agents robotics multimodal safety

arXiv AI

Jun 11

T2MM: An LLM Supported Architecture For Inquiry-Based Modeling

arXiv:2606. 11210v1 Announce Type: cross Abstract: Model Construction is a foundational practice in science learning that relies on visualization and interactivity.

By John Kos, Rudra Singh, Ashok Goel

llms multimodal

arXiv AI

Jun 11

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models

arXiv:2606. 11792v1 Announce Type: cross Abstract: Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video.

By Yuansheng Gao, Wenbin Xing, Jiahao Yuan, Kaiwen Zhou, Han Bao, Zonghui Wang, Wenzhi Chen

efficiency multimodal safety

arXiv AI

Jun 11

Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering

arXiv:2606. 11836v1 Announce Type: cross Abstract: This paper presents a novel data-free and training-free compression approach for speech foundation models using channelwise clustering via k-means.

By Haoning Xu, Zhaoqing Li, Huimeng Wang, Youjun Chen, Chengxi Deng, Mengzhe Geng, Xunying Liu

fine-tuning efficiency multimodal

arXiv AI

Jun 11

Multimodal Ordinal Modeling of Alzheimer's Disease Severity Using Structural MRI and Clinical Data

arXiv:2606. 11794v1 Announce Type: cross Abstract: Neurodegenerative diseases such as Alzheimer's disease (AD) require accurate and scalable tools for assessing disease severity, yet current clinical staging remains time-intensive and prone to variability.

By Boris-Stephan Rauchmann, Jonathan Laib, Buse Ercik, Robert Perneczky, Sergio Altares-L\'opez

multimodal

arXiv Machine Learning

Jun 11

DroneShield-AI: A Multi-Modal Sensor Fusion Framework for Real-Time Autonomous Drone Threat Detection, Behavioral Intent Classification, and Swarm Intelligence in Contested Airspace

arXiv:2606. 11687v1 Announce Type: cross Abstract: Unmanned Aerial Vehicle (UAV) threats have emerged as a defining security challenge of the 21st century.

By Marius Bayizere

agents robotics multimodal safety

arXiv AI

Jun 11

The Unreasonable Effectiveness of Discrete-Time Gaussian Process Mixtures for Robot Policy Learning

arXiv:2505. 03296v2 Announce Type: replace-cross Abstract: We present Mixture of Discrete-time Gaussian Processes (MiDiGap), a novel approach for flexible policy representation and imitation learning in robot manipulation.

By Jan Ole von Hartz, Adrian R\"ofer, Joschka Boedecker, Abhinav Valada

robotics multimodal benchmarks

arXiv AI

Jun 11

Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark

arXiv:2602. 19502v2 Announce Type: replace Abstract: Agentic AI systems are increasingly capable of autonomous data science workflows, yet clinical prediction tasks demand domain expertise that purely automated approaches struggle to provide.

By Lalitha Pranathi Pulavarthy, Raajitha Muthyala, Aravind V Kuruvikkattil, Zhenan Yin, Rashmita Kudamala, Saptarshi Purkayastha

agents multimodal benchmarks safety

arXiv AI

Jun 11

OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

arXiv:2601. 22725v4 Announce Type: replace-cross Abstract: Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck.

By Jin Li, Tao Chen, Kai Wen, Siqi Yin, Shuai Jiang, Weijie Wang, Jingwen Luo, Chenhui Wu

llms diffusion computer-vision multimodal benchmarks safety

arXiv AI

Jun 11

Pretrained self-supervised speech models can recognize unseen consonants

arXiv:2606. 11542v1 Announce Type: cross Abstract: Modern pretrained self-supervised automatic speech recognition models are trained on large-scale audio data to encode speech into contextualized representations.

By Chihiro Taguchi, \'Eric Le Ferrand, Hirosi Nakagawa, Hitomi Ono, Kanji Kato, Emily Prud'hommeaux, David Chiang

fine-tuning multimodal

arXiv AI

Jun 11

DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

arXiv:2606. 12402v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents, with an emerging strategy of scaling test-time compute to improve capability.

By Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, Ajay Sridhar, Mozhgan Nasr Azadani, Katie Luo, Clark Barrett, Jiajun Wu, Chelsea Finn, Marco Pavone

llms agents robotics multimodal

arXiv Machine Learning

Jun 11

IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents

arXiv:2606. 11652v1 Announce Type: new Abstract: This paper investigates reinforcement learning (RL) methods for improving tool-calling capabilities in multimodal small language model (SLM) agents.

By Yifan Yang, Zhen Zhang, Jiayi Tian, Liyan Tan, Zheng Zhang

llms agents nlp reinforcement-learning multimodal

arXiv Machine Learning

Jun 11

Learning Instance-Adaptive Low-Rank Orthogonal Subspaces for Clothes-Changing Person Re-Identification

arXiv:2606. 11661v1 Announce Type: cross Abstract: Clothes-changing person re-identification (CC-ReID) aims to recognize individuals despite drastic appearance changes caused by clothing variation.

By Dong-Woo Kim, Tae-Kyun Kim

llms rag multimodal benchmarks safety

arXiv AI

Jun 11

Bridging the Morphology Gap: Adapting VLA Models to Dexterous Manipulation via Intent-Conditioned Fine-Tuning

arXiv:2606. 12109v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated remarkable zero-shot generalization in robotic manipulation, yet the vast majority of pre-trained pipelines remain strictly confined to low-DoF parallel grippers.

By Chuanke Pang, Junyi Huang, Zhijun Zhao, Yaobing Wang, Kun Xu, Xilun Ding

diffusion robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 11

From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning

arXiv:2606. 11745v1 Announce Type: cross Abstract: Visual causal reasoning is essential for understanding and intervening in the physical world, requiring identification of causal variables from visual inputs and reasoning over intervention effects.

By Haoping Yu, Yuanxi Li, Jing Ma

llms multimodal

arXiv AI

Jun 11

Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding

arXiv:2606. 12047v1 Announce Type: cross Abstract: In this paper, we address the problem of zero-shot understanding of accidents from surveillance videos by identifying when an impact event occurs, what type of impact it is, and where in the frame it occurs using natural language.

By Tarandeep Singh, Soumyanetra Pal, Soham Biswas, Nishanth Chandran

llms multimodal benchmarks

arXiv AI

Jun 11

Diffusion-based Cumulative Adversarial Purification for Vision Language Models

arXiv:2506. 03933v2 Announce Type: replace-cross Abstract: Vision Language Models (VLMs) have shown remarkable capabilities in multimodal understanding, yet their susceptibility to adversarial perturbations poses a significant threat to their reliability in real-world applications.

By Jia Fu, Yongtao Wu, Yihang Chen, Kunyu Peng, Xiao Zhang, Volkan Cevher, Sepideh Pashami, Anders Holst

llms rag diffusion multimodal safety

arXiv Machine Learning

Jun 11

Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning

arXiv:2511. 14427v4 Announce Type: replace-cross Abstract: Effective contact-rich manipulation requires robots to synergistically leverage vision, force, and proprioception.

By Rickmer Krohn, Vignesh Prasad, Gabriele Tiboni, Georgia Chalvatzaki

llms rag agents reinforcement-learning robotics multimodal

arXiv AI

Jun 11

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

arXiv:2604. 13733v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation, but scaling to long-horizon tasks with sparse or imperfect rewards remains difficult due to inefficient exploration and poor credit assignment.

By Angelo Moroncelli, Roberto Zanetti, Marco Maccarini, Loris Roveda

agents reinforcement-learning robotics efficiency multimodal safety

Get the daily briefing