Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,277 stories · RSS feed

arXiv Machine Learning

Jun 15

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

arXiv:2606. 13657v2 Announce Type: replace Abstract: On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student trajectories and dense teacher supervision.

By Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jun 15

Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes

arXiv:2602. 00593v4 Announce Type: replace-cross Abstract: Despite progress on general tasks, vision-language models (VLMs) still struggle with challenges that demand both fine-grained visual grounding and external knowledge, a synergy overlooked by existing benchmarks that evaluate these abilities in isolation.

By Yifan Jiang, Cong Zhang, Bofei Zhang, Qiaofeng Zheng, Yifan Yang, Bingzhang Wang, Yew-Soon Ong

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 15

PhysVLA: Towards Physically-Grounded VLA for Embodied Robotic Manipulation

arXiv:2606. 13886v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models excel at mapping visual inputs and natural language instructions directly to robotic control policies.

By Namai Chandra, Shriram Damodaran, Lin Wang

robotics fine-tuning multimodal

arXiv AI

Jun 15

Mirage Probes: How Vision Models Fake Visual Understanding

arXiv:2606. 13870v1 Announce Type: cross Abstract: Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided.

By Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao, Raz Lapid, Amit LeVi, Allen G. Roush, Ravid Shwartz-Ziv, Hod Lipson

llms diffusion multimodal benchmarks

arXiv AI

Jun 15

Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speech

arXiv:2606. 13989v1 Announce Type: cross Abstract: Recent alignment-free non-autoregressive (NAR) text-to-speech (TTS) models formulate synthesis as a conditional infilling task, bypassing explicit duration predictors and external aligners.

By Alef Iury Siqueira Ferreira, Lucas Rafael Stefanel Gris, Luiz Fernando de Ara\'ujo Vidal, Frederico Santos de Oliveira, Christopher Dane Shulby, Anderson da Silva Soares, Arlindo Rodrigues Galv\~ao Filho

diffusion fine-tuning multimodal safety

arXiv AI

Jun 15

MirrorCheck: Efficient Adversarial Defense for Vision-Language Models

arXiv:2406. 09250v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly susceptible to sophisticated adversarial attacks, including adaptive strategies specifically designed to bypass existing defenses.

By Samar Fares, Klea Ziu, Toluwani Aremu, Nikita Durasov, Martin Tak\'a\v{c}, Pascal Fua, Ivan Laptev, Karthik Nandakumar

llms rag diffusion multimodal safety

arXiv Machine Learning

Jun 15

RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation

arXiv:2606. 14010v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have shown strong potential for end-to-end autonomous driving by jointly modeling visual perception, language reasoning, explainability and action prediction.

By Xiangyu Huang, Zhenlin Hua, Han Zhou, Shounak Sural, Ragunathan Rajkumar

agents robotics efficiency multimodal benchmarks safety

arXiv AI

Jun 15

Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models

arXiv:2606. 14647v1 Announce Type: cross Abstract: Transformer-based automatic speech recognition (ASR) models such as Whisper are highly accurate, but their predictions remain difficult to interpret.

By Ravi Ranjan, Utkarsh Grover, Xiaomin Lin, Agoritsa Polyzou

llms efficiency multimodal

arXiv AI

Jun 15

MoDiCoL: A Modular Diagnostic Continual Learning Dataset for Robust Speech Recognition

arXiv:2606. 14459v1 Announce Type: cross Abstract: Modern Automatic Speech Recognition (ASR) systems have made remarkable progress on standard benchmarks, yet performance gaps have emerged under real-world distribution shifts, caused by recording conditions, accents, speech impairments, and noise.

By Theresa Pekarek Rosin, Matthias Kerzel, Stefan Wermter

multimodal benchmarks

arXiv AI

Jun 15

How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

arXiv:2606. 13896v1 Announce Type: cross Abstract: Self-supervised geospatial foundation models (GeoFMs) learn transferable representations from remote sensing data, but their downstream behavior is difficult to characterize.

By Julia Romero, Qin Lv, Morteza Karimzadeh

llms rag computer-vision fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 15

NeST: Neuron Selective Tuning for LLM Safety

arXiv:2602. 16835v2 Announce Type: replace-cross Abstract: Safety alignment is essential for the responsible deployment of Large Language Models (LLMs).

By Sasha Behrouzi, Lichao Wu, Mohamadreza Rostami, Ahmad-Reza Sadeghi

llms fine-tuning multimodal safety

arXiv AI

Jun 15

FEMOT: Multi-Object Tracking using Frame and Event Cameras

arXiv:2606. 14094v1 Announce Type: cross Abstract: Conventional RGB cameras have been widely used in multi-object tracking due to their ability to capture rich appearance and semantic information.

By Shiao Wang, Xiao Wang, Chao Wang, Yitao Li, Menghao Liu, Bo Jiang, Yaowei Wang, Yonghong Tian, Jin Tang

multimodal benchmarks

arXiv Machine Learning

Jun 15

AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models

arXiv:2603. 18464v3 Announce Type: replace Abstract: Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models is severely bottlenecked by synchronization barriers and the high cost of environment data acquisition.

By Chengxuan Lu, Shukuan Wang, Yanjie Li, Yingying Fang, Huoyan Wang, Tian Zhang, Wei Liu, Shiji Jin, Fuyuan Qian, Peiming Li, Chao Xu, Baigui Sun, Yang Liu

reinforcement-learning robotics efficiency multimodal

arXiv AI

Jun 15

LLM-Powered AI Agent Systems and Their Applications in Industry

arXiv:2505. 16120v3 Announce Type: replace Abstract: The emergence of Large Language Models (LLMs) has reshaped agent systems.

By Guannan Liang, Qianqian Tong

llms agents efficiency multimodal

arXiv AI

Jun 15

COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

arXiv:2512. 02318v4 Announce Type: replace-cross Abstract: This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA.

By Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Mingkui Wei, Junjie Xiong

llms multimodal benchmarks

arXiv AI

Jun 15

GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models

arXiv:2606. 12817v2 Announce Type: replace Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.

By Yudong Zhang (Honor Device Co., Ltd), Lei Hu (Honor Device Co., Ltd), Daoyang Liu (The Chinese University of Hong Kong, Hong Kong, China), Jiawei Liu (Honor Device Co., Ltd), Yangfan Luo (Honor Device Co., Ltd), Zhilin Gao (Honor Device Co., Ltd), Zuojian Wang (Honor Device Co., Ltd)

llms agents multimodal benchmarks

arXiv AI

Jun 15

Interpretable Alzheimer's Diagnosis via Multimodal Fusion of Regional Brain Experts

arXiv:2512. 10966v3 Announce Type: replace-cross Abstract: Accurate and early diagnosis of Alzheimer's disease (AD) is critical for effective intervention and requires integrating complementary information from multimodal neuroimaging data.

By Farica Zhuang, Shu Yang, Dinara Aliyeva, Zixuan Wen, Duy Duong-Tran, Christos Davatzikos, Tianlong Chen, Song Wang, Li Shen

multimodal

arXiv AI

Jun 15

Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack

arXiv:2606. 14409v1 Announce Type: cross Abstract: In this report, we present Hy-Embodied-0.

By He Zhang, Lingzhu Xiang, Haitao Lin, Zeyu Huang, Minghui Wang, Dingyan Zhong, Yubo Dong, Yihao Wu, Yongming Rao, Dongsheng Zhang, Wanjia He, Ling Chen, Kai Huang, Jiahao Chen, Sichang Su, Xumin Yu, Ziyi Wang, Chengwei Zhu, Xiao Teng, Yuchun Guo, Yufeng Zhang, Yuandong Liu, Rui Wang, Zisheng Lu, Han Hu, Zhengyou Zhang

reinforcement-learning robotics fine-tuning multimodal

arXiv AI

Jun 15

ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning

arXiv:2606. 14697v1 Announce Type: cross Abstract: Building trustworthy medical multimodal large language models (MLLMs) is critical for reliable clinical decision support.

By Sicheng Yang, Hangjie Yuan, Wenjun Zhang, Jinwang Wang, Yichen Qian, Weihua Chen, Fan Wang, Lei Zhu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 15

When and How Severely: Scenario-Specific Safety Envelopes for Driving VLAs

arXiv:2606. 14238v1 Announce Type: cross Abstract: Safety certification of Vision-Language-Action (VLA) driving planners under ISO 21448 (SOTIF) rests on an Operational Design Domain (ODD) specification that answers two complementary questions: when does the planner start to fail, and how severely does it fail once it does?

By Abhinaw Priyadershi, Jelena Frtunikj

multimodal safety

Get the daily briefing