Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,069 stories · RSS feed

arXiv AI

Jun 17

RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models

arXiv:2506. 17639v2 Announce Type: replace-cross Abstract: Vision-Language-Action models (VLA) have demonstrated remarkable capabilities and strong potential in complex robotic manipulation.

By Yuxuan Chen, Yixin Han, Yize Huang, Xiao Li

reinforcement-learning robotics fine-tuning efficiency multimodal

arXiv AI

Jun 17

R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?

arXiv:2602. 03300v2 Announce Type: replace-cross Abstract: In this work, we aim to develop effective data synthesis techniques that autonomously synthesize multimodal training data for enhancing MLLMs in solving complex real-world tasks.

By Jingyi Zhang, Tianyi Lin, Huanjin Yao, Xiang Lan, Shunyu Liu, Jiaxing Huang

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 17

ProCUA-SFT Technical Report

arXiv:2606. 17321v1 Announce Type: new Abstract: Training computer-use agents (CUAs) -- models that interact with graphical desktops through screenshots and keyboard/mouse actions -- requires large-scale, diverse trajectory data collected in full desktop environments.

By Jaehun Jung, Ximing Lu, Brandon Cui, Muhammad Khalifa, Shaokun Zhang, Hao Zhang, Jin Xu, Amala Sanjay Deshmukh, Karan Sapra, Andrew Tao, Yejin Choi, Jan Kautz, Mingjie Liu, Yi Dong

agents fine-tuning multimodal

arXiv Machine Learning

Jun 17

Uncertainty Quantification for Flow-Based Vision-Language-Action Models

arXiv:2606. 18043v1 Announce Type: cross Abstract: Vision-language-action models (VLAs) combine vision-language backbones with expressive generative action heads trained via flow matching on large-scale robotic datasets.

By Ralf R\"omer, Maximilian Seeliger, Saida Liu, Ben Sturgis, Marco Bagatella, Daniel Marta, Andreas Krause, Angela P. Schoellig

diffusion robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 17

Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models

arXiv:2606. 17846v1 Announce Type: cross Abstract: Foundation models in language and multimodality achieve strong generalization by aligning heterogeneous data under a unified formulation and training at scale.

By Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, Jie Zhang, Jingyang Fan, Gengze Zhou, Qihang Peng, Chenxu Lv, Xiaoyue Chen, An Yang, Fei Huang, Junyang Lin, Dayiheng Liu, Jingren Zhou, Chenfei Wu, Xiong-Hui Chen

robotics multimodal benchmarks safety

arXiv AI

Jun 17

StepGuard: Guarding Web Navigation via Single-Step Calibration

arXiv:2606. 17871v1 Announce Type: new Abstract: Web navigation requires agents to follow natural language goals, interact with web pages, and produce accurate answers.

By Zhihao Cui, Yuchen Zhang, Xiyang Sun, Yaxiong Wang, Li Zhu, Jinpeng Hu, Liu Liu, Mengjia Li, Yujiao Wu

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jun 17

MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation

arXiv:2606. 17449v1 Announce Type: cross Abstract: While Multimodal Retrieval-Augmented Generation (M-RAG) enhances Large Vision-Language Models, it remains highly susceptible to cross-modal hallucinations, causal fabrications, and sycophancy.

By Zehang Wei, Jiaxin Dai, Jiamin Yan, Xiang Xiang

llms rag agents multimodal safety

arXiv Machine Learning

Jun 17

Multimodal Graph Negative Learning

arXiv:2606. 12863v2 Announce Type: replace Abstract: Multimodal attributed graphs (MAGs) integrate graph topology with heterogeneous modality attributes, such as text and images, thereby enabling richer modeling of complex relational systems.

By Zhengyu Wu, Xu Wang, Hongchao Qin, Xunkai Li, Guang Zeng, Rong-Hua Li, Guoren Wang

multimodal safety

arXiv AI

Jun 17

Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

arXiv:2606. 17389v1 Announce Type: cross Abstract: Multimodal Foundation Models are increasingly used as reasoning agents, making reliability, knowing when a model may hallucinate, critical.

By Logan Mann, Yi Xia, Ajit Saravanan, Ishan Dave, Saadullah Ismail, Shikhar Shiromani, Emily Huang, Ruizhe Li, Kevin Zhu

llms agents multimodal

arXiv AI

Jun 17

DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models

arXiv:2604. 24357v2 Announce Type: replace-cross Abstract: Diffusion language models generate without a fixed left-to-right order, leaving token ordering as a central algorithmic choice.

By Dake Bu, Wei Huang, Andi Han, Hau-San Wong, Qingfu Zhang, Taiji Suzuki, Atsushi Nitanda

llms diffusion reinforcement-learning multimodal

arXiv AI

Jun 17

Membership Inference Attacks against Large Audio Language Models

arXiv:2603. 28378v2 Announce Type: replace-cross Abstract: We present the first systematic Membership Inference Attack (MIA) evaluation of LALMs.

By Jia-Kai Dong, Yu-Xiang Lin, Hung-Yi Lee

llms multimodal benchmarks

arXiv AI

Jun 17

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable

arXiv:2606. 17107v1 Announce Type: cross Abstract: Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache.

By Bojie Li

agents efficiency multimodal benchmarks

arXiv AI

Jun 17

Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

arXiv:2606. 17115v1 Announce Type: cross Abstract: Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored.

By Jingyu Hu, Giuseppe Tripodi, Reed Naidoo, Sarah F. McGough, Tapabrata Chakraborti

multimodal benchmarks

arXiv AI

Jun 17

PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Space

arXiv:2606. 17924v1 Announce Type: cross Abstract: Current Vision-Language-Action (VLA) models face a trade-off between efficient action generation and explicit deliberation.

By Bochen Yang, Lianlei Shan

llms diffusion reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 17

Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

arXiv:2606. 17110v1 Announce Type: cross Abstract: Large Language Models are increasingly trained on proprietary or sensitive data, from private healthcare and financial records to user conversations containing secrets.

By Md Abdullah Al Mamun, Ngoc Phu Doan, Pedram Zaree, Ihsen Alouani, Nael Abu-Ghazaleh

llms multimodal safety

arXiv AI

Jun 17

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

arXiv:2510. 01359v2 Announce Type: replace-cross Abstract: Code-capable large language model (LLM) agents are embedded in software engineering workflows where they can read, write, and execute code, raising "jailbreak" stakes beyond text-only settings.

By Shoumik Saha, Jifan Chen, Sam Mayers, Sanjay Krishna Gouda, Zijian Wang, Varun Kumar

llms agents multimodal benchmarks safety

arXiv AI

Jun 17

When Multiple Scripts Matter: Evaluating ASR in Clinical Settings

arXiv:2606. 17826v1 Announce Type: cross Abstract: Automatic speech recognition (ASR) in non-English clinical settings is challenged by multiscript variability, where the same term may appear in multiple valid orthographic forms.

By Jean Seo, Minkyu Kim, Jeonguk Lee, Jisoo Jung, Wooseok Han, Eunho Yang

multimodal benchmarks

arXiv Machine Learning

Jun 17

SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs

arXiv:2606. 12867v2 Announce Type: replace Abstract: Multimodal-attributed graphs (MAGs) couple graph topology with node semantics from text, images, and other modalities.

By Zhengyu Wu, Xu Wang, Hongchao Qin, Xunkai Li, Guang Zeng, Rong-Hua Li, Guoren Wang

multimodal benchmarks safety

arXiv AI

Jun 17

HRDX: A Large-Scale Vector HD-Map Dataset

arXiv:2606. 17080v1 Announce Type: cross Abstract: Reliable autonomous driving requires vectorized HD maps that are geometrically accurate, semantically rich, and scalable to long-horizon driving.

By Sahith Reddy Chada, Isht Dwivedi, Nirav Savaliya

agents robotics multimodal benchmarks

Hugging Face Trending Papers

Jun 17

Multi-Modal Hyper-Graph Fusion for Low-Light Crowd Counting

Crowd counting is a fundamental task in computer vision. However, crowd counting in low-light environments remains largely underexplored, despite its practical importance in the real world.

computer-vision multimodal benchmarks

Get the daily briefing