Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,615 stories · RSS feed

arXiv AI

Jun 19

The Hidden Evolution of Disguised Visual Context inside the VLM

arXiv:2606. 20077v1 Announce Type: cross Abstract: Visual tokens enter Large Language Models (LLMs) as raw, foreign signals.

By Wish Suharitdamrong, Tony Alex, Muhammad Awais, Sara Atito

llms multimodal benchmarks

arXiv AI

Jun 19

See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View

arXiv:2606. 20045v1 Announce Type: cross Abstract: UAV Vision-Language Navigation (UAV-VLN) is typically formulated as a holistic search-and-reach problem, where long-range target discovery and final target approach are optimized and evaluated jointly.

By Fanfu Xue, En Yu, Yantian Shen, Zhikun Hu, Hongjun Wang, Yang Yang, Xindi Wang, Jiande Sun

agents robotics multimodal benchmarks safety

arXiv AI

Jun 19

UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation

arXiv:2501. 17015v2 Announce Type: replace Abstract: Simulation plays a crucial role in assessing autonomous driving systems, where the generation of realistic multi-agent behaviors is a key aspect.

By Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, Yue Wang

agents robotics multimodal benchmarks safety

arXiv AI

Jun 19

Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA

arXiv:2606. 19950v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) show great potential in medical tasks, but their elicited confidence often misaligns with actual accuracy, potentially leading to misdiagnosis or overlooking correct advice.

By Yuetian Du, Yucheng Wang, Ming Kong, Tian Liang, Qiang Long, Bingdi Chen, Qiang Zhu

llms nlp multimodal

arXiv AI

Jun 19

Cross-Dataset, Age, and Gender Generalization: A Comprehensive Analysis of Fine-Tuning Strategies for Low-Resource Children's ASR

arXiv:2606. 19791v1 Announce Type: cross Abstract: The challenge associated with recognizing dysarthric speech primarily arises from pronounced acoustic variability attributed to impaired articulatory precision.

By Paban Sapkota, Hemant Kumar Kathania, Mikko Kurimo, Sudarsana Reddy Kadiri, Shrikanth Narayanan

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

Multimodal Concept Bottleneck Models

arXiv:2606. 19882v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.

By Tongqing Shi, Ge Yan, Tuomas Oikarinen, Tsui-Wei Weng

rag multimodal benchmarks safety

arXiv AI

Jun 19

Data Standards for Humanoid Robotics: The Missing Infrastructure for Physical AI

arXiv:2606. 19769v1 Announce Type: cross Abstract: The scalability of humanoid robots will depend not only on models and hardware, but also on whether physical experience can accumulate across robots, tasks, organizations, and time.

By Shaoshan Liu, Xiugong Qin, Xuan Wu, Xuan Xia, Ning Ding, Jialu Liu, Jie Tang

robotics multimodal

arXiv AI

Jun 19

ProMUSE: Progressive Multi-modal Uncertainty-guided Staged Evidential Alzheimer Disease Classification

arXiv:2606. 19371v1 Announce Type: cross Abstract: Alzheimer's disease (AD) is a fatal disorder that destroys memory and cognitive skills in the elderly population.

By Long Doan, Branden Chen, Ethan Litton, Huan Huang, Jiajing Huang, Yixin Xie, Weihua Zhou, Nandakumar Narayanan, Chen Zhao

multimodal

arXiv Machine Learning

Jun 19

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

arXiv:2606. 20364v1 Announce Type: new Abstract: A companion study established a de-biased, cross-model VLM-as-3D-judge that reliably ranks single-image-to-3D mesh quality where cheap geometry and CLIP proxies fall short.

By Ali Asaria, Tony Salomone, Deep Gandhi

fine-tuning multimodal safety

arXiv Machine Learning

Jun 19

EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoning

arXiv:2510. 19893v2 Announce Type: replace Abstract: Medical AI systems demonstrated impressive diagnostic performance, yet they routinely show uneven accuracy across demographic groups, disadvantaging underrepresented populations.

By Shiqi Dai, Wei Dai, Jiaee Cheong, Paul Pu Liang

reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 19

GB-LSR: A Fast Local Spectral Image Representation with a Single Global Bandwidth for Continuous Reconstruction and Super-Resolution

arXiv:2606. 19617v1 Announce Type: cross Abstract: We present GB-LSR (Global-Bandwidth Local Spectral Representation), a fixed-grid local spectral representation for continuous image reconstruction.

By Max Shad, Naeem Khoshnevis

multimodal benchmarks

arXiv AI

Jun 19

TerraMind: Large-Scale Generative Multimodality for Earth Observation

arXiv:2504. 11171v5 Announce Type: replace-cross Abstract: We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO).

By Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, Nicolas Long\'ep\'e

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning

arXiv:2606. 19656v1 Announce Type: cross Abstract: A natural recipe for intelligent robotic decision-making is initializing from pretrained generative control policies, which have summarized offline experience, and adapting them to self-collected online experience.

By Calvin Luo, Chen Sun, Shuran Song

agents diffusion reinforcement-learning robotics fine-tuning multimodal

arXiv Machine Learning

Jun 19

Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

arXiv:2606. 20477v1 Announce Type: cross Abstract: We study how to train visually grounded vision-language models (VLMs) for radiology without manual spatial annotations.

By Yusuf Salcan (Computer Vision Group, University of Freiburg, Germany, CRIION-AI Lab, Freiburg, Germany), Simon Ging (Computer Vision Group, University of Freiburg, Germany, Adaptive & Agentic AI), Robin Schirrmeister (Department of Radiology, Medical Center -- University of Freiburg, Germany), Philipp Arnold (Department of Radiology, Medical Center -- University of Freiburg, Germany), Elmar Kotter (Department of Radiology, Medical Center -- University of Freiburg, Germany), Behzad Bozorgtabar (Adaptive & Agentic AI), Thomas Brox (Computer Vision Group, University of Freiburg, Germany)

llms computer-vision nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

Computational Methods and Challenges in Cell-Free DNA Analysis for Multi-Cancer Early Detection

arXiv:2606. 20174v1 Announce Type: new Abstract: Cell-free DNA (cfDNA) is a promising avenue for non-invasive multicancer early detection (MCED), in that, it can enable multiple cancer detection simultaneously from a single blood draw, with particular sensitivity to cancers that currently lack established screening programs.

By Nicko Starkey, Marcin W. Wojewodzic, Krzysztof Rzecki

multimodal safety

Hugging Face Trending Papers

Jun 18

See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View

UAV Vision-Language Navigation (UAV-VLN) is typically formulated as a holistic search-and-reach problem, where long-range target discovery and final target approach are optimized and evaluated jointly. This formulation makes it difficult to assess a critical capability of aerial embodied agents, namely whether a UAV can accurately ground a visible target and translate vision-language evidence into precise 3D motion once the target enters its field of view.

agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 18

Risk Stratification for ICU Delirium using Pervasive Ambient Sensing Information

arXiv:2606. 19292v1 Announce Type: new Abstract: Delirium is a common and serious complication in the Intensive Care Unit (ICU), associated with increased morbidity, prolonged hospital stays, and higher healthcare costs.

By Jiaqing Zhang, Sabyasachi Bandyopadhyay, Miguel Contreras, Jessica Sena, Yuanfang Ren, Andrea Davidson, Ziyuan Guan, Tezcan Ozrazgat-Baslanti, Subhash Nerella, Azra Bihorac, Parisa Rashidi

multimodal

arXiv AI

Jun 18

CAPRA: Scaling Feedback on Software Architecture Deliverables with a Multi-Agent LLM System

arXiv:2606. 18976v1 Announce Type: cross Abstract: Automated assessment in software engineering education has advanced significantly for code grading and essay scoring.

By Marco Becattini, Niccol\`o Caselli, Matteo Minin, Roberto Verdecchia, Enrico Vicario

llms agents multimodal safety

arXiv Machine Learning

Jun 18

MOLAR: Learning Multimodal Molecular Representations from Noisy Labels

arXiv:2606. 18390v1 Announce Type: new Abstract: Motivation: Noisy labels are a common challenge in molecular property prediction because molecular annotations are often obtained from assays, curated databases, or weak annotation pipelines rather than directly observed clean biological states.

By Yingxu Wang, Kunyu Zhang, Nan Yin, Yu Li, Eran Segal

multimodal benchmarks safety

arXiv AI

Jun 18

Guava: An Effective and Universal Harness for Embodied Manipulation

arXiv:2606. 18363v1 Announce Type: cross Abstract: Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents.

By Haowen Liu, Xirui Li, Shaoxiong Yao, Peng Shi, Tianyi Zhou, Jia-Bin Huang, Furong Huang, Jiayuan Mao

llms agents robotics multimodal

Get the daily briefing