Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 19

Tri-Info: Generalizable, Interpretable Failure Prediction for VLA Models via Information Theory

arXiv:2606. 19998v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models are increasingly deployed across diverse tasks, yet they remain black boxes whose physical interactions can cause irreversible harm, making generalizable and interpretable failure detection essential.

By Jinghan Yang, Yunchao Zhang, Wang Yuan, Haolun Wan, Jiaming Zhang, Zhengyang Hu, Yanchao Yang

robotics multimodal benchmarks

arXiv AI

Jun 19

A Comparative Study of Pretrained Transformer Models for Quranic ASR: Speech Representations, Label Formats, and Dataset Composition

arXiv:2606. 19747v1 Announce Type: new Abstract: Quran Automatic Speech Recognition (ASR) aims to convert Quranic recitation into text, enabling applications such as aided memorisation tools and Quranic search engines.

By Nabil Mosharraf Hossain (Greentech Apps Foundation, United Kingdom), Riasat Islam (Greentech Apps Foundation, United Kingdom, Queen Mary University of London, United Kingdom), Unaizah Obaidellah (University of Malaya, Malaysia)

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

arXiv:2606. 20118v1 Announce Type: cross Abstract: Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

By Jonghoon Lee, Seong Hyeon Park, Byungwoo Jeon, Minha Lee, Jinwoo Shin

robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 19

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

arXiv:2606. 20532v1 Announce Type: new Abstract: Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear.

By Nityanand Mathur, Hamees Sayed, Wasim Madha, Apoorv Singh, Sameer Khurana, Akshat Mandloi, Sudarshan Kamath

diffusion multimodal

arXiv AI

Jun 19

Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks

arXiv:2606. 19489v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understandable concept space.

By Ya Wang, Adrian Paschke

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jun 19

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

arXiv:2605. 20448v2 Announce Type: replace-cross Abstract: Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit?

By Animesh Maheshwari, Divyansh Sahu, Nishit Verma

llms robotics multimodal benchmarks

arXiv Machine Learning

Jun 19

Alzheimer's Disease Diagnosis using a Multimodal Approach with 3D MRI and PET

arXiv:2606. 20037v1 Announce Type: new Abstract: Alzheimer's disease (AD) is an irreversible neurodegenerative disorder and a leading cause of death worldwide.

By Loukas Ilias, Anthi-Maria Vozinaki, Christos Ntanos, Dimitris Askounis

multimodal safety

arXiv Machine Learning

Jun 19

EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoning

arXiv:2510. 19893v2 Announce Type: replace Abstract: Medical AI systems demonstrated impressive diagnostic performance, yet they routinely show uneven accuracy across demographic groups, disadvantaging underrepresented populations.

By Shiqi Dai, Wei Dai, Jiaee Cheong, Paul Pu Liang

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 19

FlowFake: Liquid Networks for Audio Deepfake Detection

arXiv:2606. 19579v1 Announce Type: cross Abstract: Audio deepfakes generated by neural text-to-speech and voice-cloning systems threaten speaker verification and public discourse at scale.

By Shivaay Dhondiyal, Divyansh Sharma, Dinesh Kumar Vishwakarma

multimodal benchmarks

arXiv AI

Jun 19

Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

arXiv:2606. 19914v1 Announce Type: cross Abstract: Art has long stood as a pivotal expression of human creativity.

By Xuetao Li, Wenke Huang, Mang Ye, Zijian Liu, Jinhua Xie, Jifeng Xuan, Miao Li

diffusion robotics fine-tuning multimodal safety

arXiv Machine Learning

Jun 19

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

arXiv:2510. 08807v2 Announce Type: replace-cross Abstract: From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities.

By Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharov, Vitor Guizilini, Yue Wang

agents robotics multimodal benchmarks

arXiv AI

Jun 19

The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation

arXiv:2603. 28387v2 Announce Type: replace Abstract: Trustworthy clinical AI requires that performance gains reflect genuine evidence integration rather than surface-level artifacts.

By Doan Nam Long Vu, Simone Balloccu

llms multimodal safety

arXiv AI

Jun 19

Hybrid Diffusion Transformer for Instruction-Guided Audio Editing via Rectified Flow

arXiv:2606. 20101v1 Announce Type: cross Abstract: Audio editing aims to modify specific content in an existing audio clip according to a natural language instruction while preserving the remaining acoustic content.

By Liting Gao, Yonggang Zhu, Yaru Chen, Dongyu Wang, Shubin Zhang, Zhenbo Li, Jean-Yves Guillemaut, Wenwu Wang

llms diffusion multimodal safety

arXiv AI

Jun 19

TerraMind: Large-Scale Generative Multimodality for Earth Observation

arXiv:2504. 11171v5 Announce Type: replace-cross Abstract: We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO).

By Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, Nicolas Long\'ep\'e

fine-tuning multimodal benchmarks

arXiv AI

Jun 19

Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Think

arXiv:2606. 20246v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models pre-trained on massive video-robot datasets have revolutionized robotic manipulation, yet their multi-billion parameter architectures impose prohibitive computational burdens during downstream fine-tuning and real-time inference.

By Gia-Binh Nguyen, Trong-Bao Ho, Thien-Loc Ha, Khoa Vo, Philip Lund M{\o}ller, Quang T. Nguyen, Long Dinh, Tuan Dam, Vu Duong, Tung M. Luu, Trung Le, Tran Nguyen Le, Minh Vu, An Thai Le, Ngan Le, Daniel Sonntag, James Zou, Jan Peters, Duy M. H. Nguyen, Ngo Anh Vien

robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jun 19

SARLO-80: Worldwide Slant SAR Language Optic Dataset 80cm

arXiv:2606. 20523v1 Announce Type: cross Abstract: Multimodal foundation models have advanced rapidly thanks to large optical benchmarks, but comparable resources for synthetic aperture radar (SAR) remain limited.

By Sol\`ene Debuys\`ere, Nicolas Trouv\'e, Nathan Letheule, Elise Colin, Georgia Channing

multimodal benchmarks safety

arXiv AI

Jun 19

Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

arXiv:2606. 20470v1 Announce Type: cross Abstract: Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents.

By Reza Soosahabi, Vivek Namsani

agents multimodal benchmarks safety

arXiv Machine Learning

Jun 19

Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning

arXiv:2606. 19823v1 Announce Type: cross Abstract: Automatic speech recognition remains unreliable for dysarthric speech due to data scarcity and high inter-speaker variability.

By Satwinder Singh, Qianli Wang, Zihan Zhong, Clarion Mendes, Hasegawa-Johnson, Waleed Abdulla, Seyed Reza Shahamiri

fine-tuning multimodal

arXiv AI

Jun 19

Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs

arXiv:2606. 20177v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in various Remote Sensing (RS) tasks.

By Haochen Han, Jue Wang, Alex Jinpeng Wang, Fangming Liu

llms multimodal benchmarks safety

arXiv AI

Jun 19

UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation

arXiv:2501. 17015v2 Announce Type: replace Abstract: Simulation plays a crucial role in assessing autonomous driving systems, where the generation of realistic multi-agent behaviors is a key aspect.

By Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, Yue Wang

agents robotics multimodal benchmarks safety

Get the daily briefing