Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,414 stories · RSS feed

arXiv AI

Jun 2

Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training

arXiv:2601. 23220v2 Announce Type: replace-cross Abstract: Despite recent Multimodal Large Language Models (MLLMs)' linguistic prowess in medical diagnosis, we find even state-of-the-art MLLMs suffer from a critical perceptual deficit: geometric blindness.

By Anglin Liu, Ruichao Chen, Yi Lu, Hongxia Xu, Jintai Chen

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

Spatial Representation Learning Beyond Pixels: Unifying Raster Data and Vector Semantics for Human-Centric Geospatial Foundation Models

arXiv:2606. 02374v1 Announce Type: new Abstract: Earth Observation (EO) has fundamentally transformed the monitoring of environmental processes and human activities up to planetary scale.

By Steffen Knoblauch, Hao Li, Gengchen Mai, Konstantin Klemmer, Song Gao, WenWen Li

rag multimodal

arXiv AI

Jun 2

Token Predictors Are Not Planners: Building Physically Grounded Causal Reasoners

arXiv:2606. 01810v1 Announce Type: new Abstract: Current benchmarks for embodied vision-language planning often favor linguistic next-token prediction over physically grounded next-state reasoning.

By Zheng Lu, Mingqi Gao, Qinlei Xie, Wanqi Zhong, Hanwen Cui, Heng Cao, Zirui Song, Yifan Yang, Chong Luo, Bei Liu, Yiming Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 2

InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning

arXiv:2509. 12263v3 Announce Type: replace Abstract: Large multimodal models (LMMs) encode physical laws observed during training, such as momentum conservation, as parametric knowledge.

By Gautam Sreekumar, Vishnu Naresh Boddeti

agents nlp multimodal benchmarks safety

arXiv AI

Jun 2

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

arXiv:2606. 00054v1 Announce Type: cross Abstract: Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models.

By Zhiyuan Feng, Qixiu Li, Huizhi Liang, Rushuai Yang, Yichao Shen, Zhiying Du, Zhaowei Zhang, Yu Deng, Li Zhao, Hao Zhao, Zongqing Lu, Oier Mees, Marc Pollefeys, Jiaolong Yang, Baining Guo

robotics multimodal

arXiv AI

Jun 2

Ryze: Evidence-Enriched Data Synthesis from Biomedical Papers

arXiv:2606. 00902v1 Announce Type: new Abstract: General-purpose VLMs remain unreliable for biomedical research because valid answers in scientific papers depend on evidence split across figures, tables, charts, captions, and referring text.

By Yeqi Huang, Yue Chen, Yanwei Ye, Guanhao Su, Luo Mai

llms computer-vision reinforcement-learning fine-tuning multimodal

arXiv Machine Learning

Jun 2

Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

arXiv:2601. 11460v2 Announce Type: replace-cross Abstract: Learning structured task representations from human demonstrations is essential for bimanual manipulation, where action ordering, object involvement, and interaction geometry vary significantly across executions.

By Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

llms robotics fine-tuning multimodal

arXiv AI

Jun 2

SilentDrift: Exploiting Action Chunking for Stealthy Backdoor Attacks on Vision-Language-Action Models

arXiv:2601. 14323v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are increasingly deployed in safety-critical robotic applications, yet their security vulnerabilities remain underexplored.

By Bingxin Xu, Yuzhang Shang, Binghui Wang, Emilio Ferrara

rag robotics multimodal safety

arXiv AI

Jun 2

ES-Merging: Biological MLLM Merging via Embedding Space Signals

arXiv:2603. 14405v2 Announce Type: replace-cross Abstract: Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery.

By Wonbin Lee, Dongki Kim, Sung Ju Hwang

llms rag multimodal

arXiv AI

Jun 2

DraDDP: A Multimodal Multi-Party Dialogue Discourse Parsing Dataset

arXiv:2606. 00012v1 Announce Type: cross Abstract: Multi-party dialogue discourse parsing aims to identify dependency structures and relation types between utterances in conversations.

By Shannan Liu, Peifeng Li, Yaxin Fan, Qiaoming Zhu

multimodal benchmarks

arXiv Machine Learning

Jun 2

Prior-Guided Multi-Omic Transformers for Single-Cell Gene Regulatory Network Inference

arXiv:2606. 00685v1 Announce Type: new Abstract: Gene regulatory networks (GRNs) capture transcription factor-target interactions and are central to understanding cell-state regulation and disease.

By Tianyang Xu, Tianci Liu, Niraj Rayamajhi, Ryan Patrick, Kranthi Varala, Ying Li, Jing Gao

llms multimodal

arXiv Machine Learning

Jun 2

UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

arXiv:2510. 13774v2 Announce Type: replace Abstract: Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data.

By Dominik J. M\"uhlematter, Lin Che, Ye Hong, Martin Raubal, Nina Wiedemann

llms multimodal benchmarks

arXiv AI

Jun 2

MURMUR: An Efficient Inference System for Long-Form ASR

arXiv:2606. 01483v1 Announce Type: cross Abstract: Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two.

By Wei-Tzu Lee, Keisuke Kamahori, Baris Kasikci

efficiency multimodal

arXiv AI

Jun 2

AtomEval: Validity-Aware Atomic Evaluation of Adversarial Claim Rewriting in Fact Verification

arXiv:2604. 07967v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can rewrite refuted claims to evade evidence-based fact verifiers, but conventional attack success rate (ASR) can be inflated when rewrites change, weaken, or correct the false proposition they are supposed to preserve.

By Hongyi Cen, Mingxin Wang, Yule Liu, Jingyi Zheng, Hanze Jia, Tan Tang

llms multimodal safety

arXiv Machine Learning

Jun 2

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

arXiv:2603. 11653v2 Announce Type: replace Abstract: Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments.

By Jiaheng Hu, Jay Shim, Chen Tang, Yoonchang Sung, Bo Liu, Peter Stone, Roberto Martin-Martin

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

arXiv:2605. 23500v2 Announce Type: replace-cross Abstract: Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis.

By Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms agents computer-vision reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 2

Towards Sparse Video Understanding and Reasoning

arXiv:2602. 13602v2 Announce Type: replace-cross Abstract: We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA).

By Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

llms agents nlp fine-tuning multimodal benchmarks

arXiv AI

Jun 2

Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

arXiv:2601. 04946v3 Announce Type: replace-cross Abstract: Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering.

By Subhadeep Roy, Gagan Bhatia, Steffen Eger

rag diffusion multimodal benchmarks safety

arXiv AI

Jun 2

Detect Before You Leap: Mirage Detection in Vision-Language Models

arXiv:2606. 00435v1 Announce Type: cross Abstract: Vision-language models (VLMs) can produce confident visual answers even when the required visual evidence is missing, blank, or unrelated to the question.

By Sayeed Shafayet Chowdhury, Md. Shaown Miah

llms rag nlp multimodal safety

arXiv Machine Learning

Jun 2

GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning

arXiv:2605. 03403v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has recently shown strong performance in post-training large language models and vision-language models.

By Yujun Li, Hongyuan Zhang, Yuan Yuan

llms reinforcement-learning multimodal benchmarks safety

Get the daily briefing