Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 19

Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

arXiv:2512. 20014v3 Announce Type: replace-cross Abstract: While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring my cup," where the robot must act on one specific instance among visually similar objects.

By Sangoh Lee, Sangwoo Mo, Wook-Shin Han

llms rag robotics multimodal benchmarks

arXiv AI

Jun 19

Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

arXiv:2606. 19914v1 Announce Type: cross Abstract: Art has long stood as a pivotal expression of human creativity.

By Xuetao Li, Wenke Huang, Mang Ye, Zijian Liu, Jinhua Xie, Jifeng Xuan, Miao Li

diffusion robotics fine-tuning multimodal safety

arXiv Machine Learning

Jun 19

Multimodal Concept Bottleneck Models

arXiv:2606. 19882v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.

By Tongqing Shi, Ge Yan, Tuomas Oikarinen, Tsui-Wei Weng

rag multimodal benchmarks safety

arXiv AI

Jun 19

ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models

arXiv:2606. 19965v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts.

By Yihao Wang, Zijian He, Jie Ren, Keze Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 19

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

arXiv:2606. 20364v1 Announce Type: new Abstract: A companion study established a de-biased, cross-model VLM-as-3D-judge that reliably ranks single-image-to-3D mesh quality where cheap geometry and CLIP proxies fall short.

By Ali Asaria, Tony Salomone, Deep Gandhi

fine-tuning multimodal safety

arXiv AI

Jun 19

Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks

arXiv:2606. 19489v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understandable concept space.

By Ya Wang, Adrian Paschke

llms rag multimodal benchmarks safety

arXiv AI

Jun 19

REVEAL++: Differentiable Phenotypic Grouping for Vision-Language Retinal Modeling of Alzheimer's Disease Risk

arXiv:2606. 19522v1 Announce Type: new Abstract: The retina offers a noninvasive window into neurodegenerative disease, capturing subtle structural patterns associated with a risk of future cognitive decline.

By Ethan Elio Meidinger, Seowung Leem, Zeyun Zhao, Ruogu Fang

rag multimodal safety

arXiv AI

Jun 19

The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation

arXiv:2603. 28387v2 Announce Type: replace Abstract: Trustworthy clinical AI requires that performance gains reflect genuine evidence integration rather than surface-level artifacts.

By Doan Nam Long Vu, Simone Balloccu

llms multimodal safety

arXiv AI

Jun 19

Hybrid Diffusion Transformer for Instruction-Guided Audio Editing via Rectified Flow

arXiv:2606. 20101v1 Announce Type: cross Abstract: Audio editing aims to modify specific content in an existing audio clip according to a natural language instruction while preserving the remaining acoustic content.

By Liting Gao, Yonggang Zhu, Yaru Chen, Dongyu Wang, Shubin Zhang, Zhenbo Li, Jean-Yves Guillemaut, Wenwu Wang

llms diffusion multimodal safety

arXiv Machine Learning

Jun 19

DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning

arXiv:2606. 19656v1 Announce Type: cross Abstract: A natural recipe for intelligent robotic decision-making is initializing from pretrained generative control policies, which have summarized offline experience, and adapting them to self-collected online experience.

By Calvin Luo, Chen Sun, Shuran Song

agents diffusion reinforcement-learning robotics fine-tuning multimodal

arXiv Machine Learning

Jun 19

Bioacoustic Geolocation: Species Sounds as Geographic Signals

arXiv:2505. 18726v3 Announce Type: replace-cross Abstract: Can we determine someone's geographic location solely from the sounds they hear?

By Mustafa Chasmai, Wuao Liu, Subhransu Maji, Grant Van Horn

multimodal benchmarks

arXiv AI

Jun 19

ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

arXiv:2606. 20280v1 Announce Type: cross Abstract: Leveraging Multimodal Large Language Models (MLLMs) via contrastive learning has become a mainstream paradigm for improving the performance of Universal Multimodal Retrieval (UMR).

By Yuhan Liu, Pei Fu, Hang Li, Yukun Qi, Chao Jiang, Jingwen Fu, Zhen Liu, Bin Qin, Zhenbo Luo, Jian Luan, Jingmin Xin

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 19

A deep learning framework for jointly solving transient Fokker-Planck equations with arbitrary parameters and initial distributions

arXiv:2604. 06001v2 Announce Type: replace-cross Abstract: Efficiently solving the Fokker-Planck equation (FPE) is central to analyzing complex parameterized stochastic systems.

By Xiaolong Wang, Jing Feng, Qi Liu, Chengli Tan, Yuanyuan Liu, Yong Xu

multimodal

arXiv AI

Jun 19

A Comparative Study of Pretrained Transformer Models for Quranic ASR: Speech Representations, Label Formats, and Dataset Composition

arXiv:2606. 19747v1 Announce Type: new Abstract: Quran Automatic Speech Recognition (ASR) aims to convert Quranic recitation into text, enabling applications such as aided memorisation tools and Quranic search engines.

By Nabil Mosharraf Hossain (Greentech Apps Foundation, United Kingdom), Riasat Islam (Greentech Apps Foundation, United Kingdom, Queen Mary University of London, United Kingdom), Unaizah Obaidellah (University of Malaya, Malaysia)

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 19

Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving

arXiv:2606. 20274v1 Announce Type: new Abstract: Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories.

By Shihao Ji, HongXi Li, Zihui Song, Mingyu Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 19

ProMUSE: Progressive Multi-modal Uncertainty-guided Staged Evidential Alzheimer Disease Classification

arXiv:2606. 19371v1 Announce Type: cross Abstract: Alzheimer's disease (AD) is a fatal disorder that destroys memory and cognitive skills in the elderly population.

By Long Doan, Branden Chen, Ethan Litton, Huan Huang, Jiajing Huang, Yixin Xie, Weihua Zhou, Nandakumar Narayanan, Chen Zhao

multimodal

arXiv AI

Jun 19

Frequency-Aware Flow Matching for Continuous and Consistent Robotic Action Generation

arXiv:2606. 20135v1 Announce Type: cross Abstract: Flow matching has emerged as a standard paradigm for robotic manipulation owing to its strong expressive power for modelling complex, multimodal action distributions, alongside similar approaches like diffusion policy.

By Jianing Guo, Fangzheng Chen, Zihao Mao, Wong Lik Hang Kenny, Zhenhong Wu, Yu Li, Yishuai Cai, Yuanpei Chen, Yikun Ban, Kai Chen, Qi Dou, Yaodong Yang, Xianglong Liu, Huijie Zhao, Simin Li

diffusion robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 19

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

arXiv:2605. 20448v2 Announce Type: replace-cross Abstract: Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit?

By Animesh Maheshwari, Divyansh Sahu, Nishit Verma

llms robotics multimodal benchmarks

Hugging Face Trending Papers

Jun 18

See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View

UAV Vision-Language Navigation (UAV-VLN) is typically formulated as a holistic search-and-reach problem, where long-range target discovery and final target approach are optimized and evaluated jointly. This formulation makes it difficult to assess a critical capability of aerial embodied agents, namely whether a UAV can accurately ground a visible target and translate vision-language evidence into precise 3D motion once the target enters its field of view.

agents robotics multimodal benchmarks safety

arXiv AI

Jun 18

WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

arXiv:2604. 06367v2 Announce Type: replace-cross Abstract: Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries.

By Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, Kassem Fawaz

llms agents multimodal benchmarks safety

Get the daily briefing