Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv Machine Learning

Jun 2

Modeling Robotics Dataset Construction as an Artifact-Based Build Process

arXiv:2606. 00162v1 Announce Type: cross Abstract: Robotic systems generate large volumes of multimodal sensor data, but converting ROS bag recordings into machine learning datasets is often handled by ad hoc sequential scripts, creating engineering overhead and slow iteration cycles.

By Leon Pohl, Lukas Beer, George Sebastian, Mirko Maehlisch

robotics multimodal

arXiv Machine Learning

Jun 2

HOIST: Humanoid Optimization with Imitation and Sample-efficient Tuning for Manipulating Suspended Loads

arXiv:2606. 00252v1 Announce Type: cross Abstract: Manipulating suspended payloads with humanoid robots is challenging because the robot can only influence an underactuated, oscillatory load through whole-body motion and intermittent contact.

By Songyang Liu, Shunyu Yao, Dingyuan Huang, Shuai Li

reinforcement-learning robotics multimodal

arXiv Machine Learning

Jun 2

Single-Channel Tissue Segmentation via Cross-Modal Distillation from Foundation Models

arXiv:2606. 00928v1 Announce Type: cross Abstract: Multiplexed fluorescence microscopy improves tissue segmentation by providing complementary channels including nuclear (DAPI) and membrane (E-cadherin), that together encode richer spatial context than single-channel imaging alone.

By Sakib Mohammad, Jarin Ritu, Md Sakhawat Hossain

computer-vision efficiency multimodal

arXiv AI

Jun 2

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

arXiv:2606. 00054v1 Announce Type: cross Abstract: Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models.

By Zhiyuan Feng, Qixiu Li, Huizhi Liang, Rushuai Yang, Yichao Shen, Zhiying Du, Zhaowei Zhang, Yu Deng, Li Zhao, Hao Zhao, Zongqing Lu, Oier Mees, Marc Pollefeys, Jiaolong Yang, Baining Guo

robotics multimodal

arXiv Machine Learning

Jun 2

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

arXiv:2606. 01207v1 Announce Type: cross Abstract: The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding.

By Zhiqiang Zhou, Xuezhen Xie

llms multimodal safety

arXiv AI

Jun 2

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.

By Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

llms multimodal benchmarks safety

arXiv AI

Jun 2

Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training

arXiv:2601. 23220v2 Announce Type: replace-cross Abstract: Despite recent Multimodal Large Language Models (MLLMs)' linguistic prowess in medical diagnosis, we find even state-of-the-art MLLMs suffer from a critical perceptual deficit: geometric blindness.

By Anglin Liu, Ruichao Chen, Yi Lu, Hongxia Xu, Jintai Chen

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

MMSkills: Towards Multimodal Skills for General Visual Agents

arXiv:2605. 13527v3 Announce Type: replace Abstract: Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines.

By Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

agents multimodal benchmarks

arXiv AI

Jun 2

Property Prediction of Stacked Bilayer Materials: A Multimodal Learning Approach

arXiv:2606. 01012v1 Announce Type: new Abstract: AI for materials science is a critical topic within AI for science, aiming to accelerate materials discovery and produce accurate property predictions.

By An Vuong, Minh-Hao Van, Chen Zhao, Xintao Wu

efficiency multimodal

arXiv Machine Learning

Jun 2

Adversarial Dual On-Policy Distillation from Expressive Teacher

arXiv:2605. 27095v2 Announce Type: replace Abstract: Learning from demonstrations in embodied control is often cast as behavioral cloning, and recent diffusion or flow-matching policies improve this paradigm by modeling multi-modal expert actions.

By Zhenglin Wan, Jingxuan Wu, Xingrui Yu, Chubin Zhang, Mingcong Lei, Bo An, Ivor W. Tsang, Yang You

diffusion robotics efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Domain Adaptation with a Single Vision-Language Embedding

arXiv:2410. 21361v2 Announce Type: replace-cross Abstract: Domain adaptation has been extensively investigated in computer vision but still requires access to target data at the training time, which might be difficult to obtain in real-world autonomous driving scenarios, especially under rare or adverse conditions.

By Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick P\'erez, Raoul de Charette

rag agents computer-vision robotics fine-tuning multimodal

arXiv AI

Jun 2

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

arXiv:2606. 01393v1 Announce Type: cross Abstract: Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems.

By Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo, Zhenting Qi, Konwoo Kim, Longtian Ye, Xiaolong Luo, Jinhe Bi, Henry Zhang, Haris Riaz, Xuan Zhang, Yunze Xiao, Bangya Liu, Tom Tang, Yunfei Zhao, Qunshu Lin, Zihan Wang, Minghao Liu, Michael Lingzhi Li, Yilun Du, Jesse Thomason, Rogerio Feris, Alex Pentland, Zexue He

llms computer-vision multimodal benchmarks

arXiv AI

Jun 2

DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

arXiv:2606. 01277v1 Announce Type: cross Abstract: Current end-to-end autonomous driving systems predominantly rely on frame-based sensors, which suffer from inherent perception latency and motion blur during highly dynamic encounters, specifically sudden pedestrian crossings.

By Oskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translation

arXiv:2505. 18614v5 Announce Type: replace-cross Abstract: Lyrics translation requires both accurate semantic transfer and preservation of musical rhythm, syllabic structure, and poetic style.

By Woohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae Yu

llms multimodal benchmarks safety

arXiv AI

Jun 2

Beyond Task Success: Behavioral and Representational Diagnostics for WAM and VLA

arXiv:2606. 01095v1 Announce Type: cross Abstract: Vision-language-action (VLA) policies and World-Action Models (WAM) represent two increasingly important paradigms for robotic manipulation.

By Hung Mai, Bin Zhu, Tuan Do

robotics multimodal

arXiv AI

Jun 2

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

arXiv:2606. 01057v1 Announce Type: cross Abstract: Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack.

By Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, Jindong Chen

llms agents multimodal benchmarks

arXiv AI

Jun 2

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

arXiv:2606. 01016v1 Announce Type: cross Abstract: While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription.

By Sicheng Yang, Shulan Ruan, Shiwei Wu, Yu Liu, Lu Fan, Zhi Li, You He

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies

arXiv:2508. 20072v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models adapt large vision-language backbones to map images and instructions into robot actions.

By Zhixuan Liang, Yizhuo Li, Tianshuo Yang, Chengyue Wu, Sitong Mao, Liuao Pei, Tian Nian, Shunbo Zhou, Xiaokang Yang, Jiangmiao Pang, Yao Mu, Ping Luo

llms diffusion robotics multimodal

arXiv AI

Jun 2

Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion

arXiv:2606. 00299v1 Announce Type: cross Abstract: While Video Diffusion Models (VDMs) excel at synthesizing high-fidelity videos, enabling precise camera and scene control remains challenging.

By Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos

diffusion fine-tuning multimodal

arXiv Machine Learning

Jun 2

PaCX-MAE: Physiology-Augmented Chest X-Ray Masked Autoencoder

arXiv:2606. 01537v1 Announce Type: cross Abstract: Clinical diagnosis often requires combining imaging with physiological measurements, yet deployed models typically operate on unimodal data.

By Yancheng Liu, Kenichi Maeda, Manan Pancholy

rag computer-vision efficiency multimodal benchmarks

Get the daily briefing