Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,277 stories · RSS feed

arXiv Machine Learning

Jun 19

EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoning

arXiv:2510. 19893v2 Announce Type: replace Abstract: Medical AI systems demonstrated impressive diagnostic performance, yet they routinely show uneven accuracy across demographic groups, disadvantaging underrepresented populations.

By Shiqi Dai, Wei Dai, Jiaee Cheong, Paul Pu Liang

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 19

PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

arXiv:2606. 19534v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks.

By Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, Tao Zhang, Jacky Mai, Yihan Wang, Haochen Wang, Jinbin Bai, Ling Yang, Yunhai Tong

llms diffusion multimodal benchmarks

arXiv AI

Jun 19

Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation

arXiv:2606. 19797v1 Announce Type: cross Abstract: Dysarthric speech recognition is crucial for facilitating effective communication among individuals with dysarthria.

By Paban Sapkota, Hemant Kumar Kathania, Sudarsana Reddy Kadiri, Shrikanth Narayanan

fine-tuning multimodal

arXiv Machine Learning

Jun 19

Towards Modality-imbalanced Federated Graph Learning: A Data Synthesis-based Approach

arXiv:2606. 20382v1 Announce Type: new Abstract: MultiModal Federated Graph Learning (MM-FGL) offers a natural collaborative training paradigm, but its practical deployment is challenged by two granularities of modality imbalance.

By Zhengyu Wu, Hongchao Qin, Xunkai Li, Zekai Chen, Rong-Hua Li, Guoren Wang

multimodal safety

arXiv Machine Learning

Jun 19

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

arXiv:2510. 08807v2 Announce Type: replace-cross Abstract: From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities.

By Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharov, Vitor Guizilini, Yue Wang

agents robotics multimodal benchmarks

arXiv AI

Jun 19

Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Think

arXiv:2606. 20246v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models pre-trained on massive video-robot datasets have revolutionized robotic manipulation, yet their multi-billion parameter architectures impose prohibitive computational burdens during downstream fine-tuning and real-time inference.

By Gia-Binh Nguyen, Trong-Bao Ho, Thien-Loc Ha, Khoa Vo, Philip Lund M{\o}ller, Quang T. Nguyen, Long Dinh, Tuan Dam, Vu Duong, Tung M. Luu, Trung Le, Tran Nguyen Le, Minh Vu, An Thai Le, Ngan Le, Daniel Sonntag, James Zou, Jan Peters, Duy M. H. Nguyen, Ngo Anh Vien

robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jun 19

VCG: A Multimodal Retrieval Framework for E-Commerce Video Feeds under Extreme Cold-Start Conditions

arXiv:2606. 19627v1 Announce Type: cross Abstract: The digital commerce landscape is shifting from static, search-driven catalogs to dynamic, immersive video feeds.

By Katya Mirylenka, Egor Malykh, Mahdyar Ravanbakhsh, Michael Gygli, Marco-Andrea Buchmann, Andrew Dzhoha, Svitlana Borzenko, Francesca Catino, Mohamed Gaafar, Maarten Versteegh, Thomas Kober, Dario d'Andrea, Ellie Langhans

llms rag multimodal

arXiv AI

Jun 19

HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-trainin

arXiv:2606. 20189v1 Announce Type: cross Abstract: Leveraging Vision Foundation Models (VFMs) for camera-to-LiDAR knowledge distillation offers a promising solution to the scarcity of annotated data needed to represent the immense geometric and kinematic diversity of real-world autonomous driving (AD).

By Maciej Wozniak, Jesper Ericsson, Hariprasath Govindarajan, Truls Nyberg, Thomas Gustafsson, Patric Jensfelt, Olov Andersson

agents diffusion computer-vision robotics efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 19

A deep learning framework for jointly solving transient Fokker-Planck equations with arbitrary parameters and initial distributions

arXiv:2604. 06001v2 Announce Type: replace-cross Abstract: Efficiently solving the Fokker-Planck equation (FPE) is central to analyzing complex parameterized stochastic systems.

By Xiaolong Wang, Jing Feng, Qi Liu, Chengli Tan, Yuanyuan Liu, Yong Xu

multimodal

arXiv Machine Learning

Jun 19

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

arXiv:2605. 20448v2 Announce Type: replace-cross Abstract: Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit?

By Animesh Maheshwari, Divyansh Sahu, Nishit Verma

llms robotics multimodal benchmarks

arXiv Machine Learning

Jun 19

Bioacoustic Geolocation: Species Sounds as Geographic Signals

arXiv:2505. 18726v3 Announce Type: replace-cross Abstract: Can we determine someone's geographic location solely from the sounds they hear?

By Mustafa Chasmai, Wuao Liu, Subhransu Maji, Grant Van Horn

multimodal benchmarks

arXiv Machine Learning

Jun 19

DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning

arXiv:2606. 19656v1 Announce Type: cross Abstract: A natural recipe for intelligent robotic decision-making is initializing from pretrained generative control policies, which have summarized offline experience, and adapting them to self-collected online experience.

By Calvin Luo, Chen Sun, Shuran Song

agents diffusion reinforcement-learning robotics fine-tuning multimodal

arXiv Machine Learning

Jun 19

Computational Methods and Challenges in Cell-Free DNA Analysis for Multi-Cancer Early Detection

arXiv:2606. 20174v1 Announce Type: new Abstract: Cell-free DNA (cfDNA) is a promising avenue for non-invasive multicancer early detection (MCED), in that, it can enable multiple cancer detection simultaneously from a single blood draw, with particular sensitivity to cancers that currently lack established screening programs.

By Nicko Starkey, Marcin W. Wojewodzic, Krzysztof Rzecki

multimodal safety

arXiv AI

Jun 19

Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

arXiv:2512. 20014v3 Announce Type: replace-cross Abstract: While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring my cup," where the robot must act on one specific instance among visually similar objects.

By Sangoh Lee, Sangwoo Mo, Wook-Shin Han

llms rag robotics multimodal benchmarks

arXiv AI

Jun 19

Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA

arXiv:2606. 19950v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) show great potential in medical tasks, but their elicited confidence often misaligns with actual accuracy, potentially leading to misdiagnosis or overlooking correct advice.

By Yuetian Du, Yucheng Wang, Ming Kong, Tian Liang, Qiang Long, Bingdi Chen, Qiang Zhu

llms nlp multimodal

arXiv Machine Learning

Jun 19

Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations

arXiv:2606. 19951v1 Announce Type: cross Abstract: Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear.

By Masato Takagi, Masaya Kawamura, Reo Shimizu, Yuma Shirahata

robotics multimodal

arXiv AI

Jun 19

CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation

arXiv:2605. 10873v2 Announce Type: replace-cross Abstract: Recovering editable CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existing evaluations are fragmented across datasets, modalities, and metrics.

By Anna C. Doris, Jacob Thomas Sony, Ghadi Nehme, Era Syla, Amin Heyrani Nobari, Faez Ahmed

multimodal benchmarks

arXiv AI

Jun 19

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

arXiv:2606. 20532v1 Announce Type: new Abstract: Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear.

By Nityanand Mathur, Hamees Sayed, Wasim Madha, Apoorv Singh, Sameer Khurana, Akshat Mandloi, Sudarshan Kamath

diffusion multimodal

arXiv AI

Jun 19

SleepMaMi: A Universal Sleep Foundation Model for Integrating Macro- and Micro-structures

arXiv:2602. 07628v2 Announce Type: replace Abstract: While the shift toward unified foundation models has revolutionized many deep learning domains, sleep medicine remains largely restricted to task-specific models that focus on localized micro-structure features.

By Keondo Park, Younghoon Na, Yourim Choi, Hyunwoo Ryu, Hyun-Woo Shin, Hyung-Sin Kim

multimodal benchmarks

arXiv Machine Learning

Jun 19

Online Dynamic Batching with Formal Guarantees for LLM Training

arXiv:2606. 19989v1 Announce Type: cross Abstract: Modern LLM training breaks a core assumption behind offline batch samplers: the true training cost of a sample is only observable after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion.

By Dian Li, Zekun Wang, Yaoru Wang, Jiahong Yan

llms nlp fine-tuning efficiency multimodal safety

Get the daily briefing