Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,022 stories · RSS feed

arXiv AI

Jul 21

ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

arXiv:2607. 17884v1 Announce Type: new Abstract: Vision Language Models (VLMs) achieve strong reasoning with Chain-of-Thought (CoT) prompting but incur high sequential-generation cost, error accumulation, and limited self-correction.

By Keuntae Kim, Beomseok Lee, Hyunwoo Kim, Yong Suk Choi

llms diffusion multimodal benchmarks

arXiv AI

Jul 21

HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization

arXiv:2607. 17994v1 Announce Type: cross Abstract: Video understanding has become more and more important with the growth of Artificial Intelligence (AI) for video generation.

By Rui Chu, Yingjie Lao

llms diffusion nlp multimodal benchmarks

arXiv AI

Jul 21

From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence

arXiv:2607. 16560v1 Announce Type: new Abstract: We propose a language representation for multimodal data in which any observation, whether image, video, or text, is expressed as a bag of atomic propositions, simple statements about the entities, actions, and relations in a scene.

By Nadine Chang, Maying Shen, Shizhe Diao, Jialiang Wang, Jingde Chen, Thomas Breuel, Pavlo Molchanov, Rafid Mahmood, Jose M. Alvarez

agents robotics multimodal safety

arXiv Machine Learning

Jul 21

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

arXiv:2607. 16789v1 Announce Type: new Abstract: Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities.

By Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

COLIP-2: Olfaction-Vision-Language Embeddings

arXiv:2607. 17559v1 Announce Type: cross Abstract: The Contrastive Olfaction-Language-Image Pre-training 2 (COLIP-2) model is a multimodal embeddings space that places olfaction as a first-class citizen among vision and language.

By Kordel Kade France

rag robotics multimodal

arXiv Machine Learning

Jul 21

Patch Policy: Efficient Embodied Control via Dense Visual Representations

arXiv:2607. 18236v1 Announce Type: cross Abstract: Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning.

By Gaoyue Zhou, Zichen Jeff Cui, Ada Langford, Bowen Tan, Yann LeCun, Lerrel Pinto

llms robotics fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 21

Multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis under unseen working conditions

arXiv:2512. 24679v2 Announce Type: replace Abstract: Intelligent fault diagnosis has become an indispensable technique for ensuring machinery reliability.

By Pengcheng Xia, Yixiang Huang, Chengjin Qin, Chengliang Liu

fine-tuning multimodal benchmarks

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

arXiv AI

Jul 21

Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA

arXiv:2607. 17834v1 Announce Type: cross Abstract: Endoscopic visual question answering (VQA) increasingly asks complex questions that combine several endoscopic answer components rather than isolated factual queries.

By Yuhao Liu, Cheng Zhao, Guanghui Yue

nlp multimodal benchmarks

arXiv AI

Jul 21

AdaSurvMamba: Dynamic Fusion and Semantic Scanning for Multimodal Survival Analysis

arXiv:2607. 16260v1 Announce Type: cross Abstract: Multimodal survival analysis utilizing whole slide images (WSIs) and genomic profiles is fundamental for cancer prognosis.

By Jialong Zhong, Tingwei Liu, Baokun Yue, Jingjing Li, Yongri Piao, Miao Zhang, Leiye Liu, Jiahong Jiang, Wei Ji, Huchuan Lu

multimodal

arXiv AI

Jul 21

Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation

arXiv:2607. 16303v1 Announce Type: cross Abstract: Medical Vision-Language Models (Med-VLMs) require reliable reasoning from fine-grained visual evidence, yet existing models can produce plausible clinical answers by relying on language priors or medical templates rather than truly attending to diagnosis-critical regions.

By Yunhang Qian, Jiaquan Yu, Jiawei Liu, Meng Wang, Hongwei Bran Li, Xiaobin Hu

llms fine-tuning efficiency multimodal safety

arXiv AI

Jul 21

DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks

arXiv:2511. 14592v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) show great promise for autonomous driving, but their suitability for safety-critical scenarios is largely unexplored, raising safety concerns.

By Xianhui Meng, Yuchen Zhang, Zhijian Huang, Zheng Lu, Ziling Ji, Yandan Lin, Yaoyao Yin, Hongyuan Zhang, Wei Zhou, Guangfeng Jiang, Li Zhang, Long Chen, Hangjun Ye, Jun Liu, Xiaoshuai Hao

llms agents robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

arXiv:2603. 26648v3 Announce Type: replace-cross Abstract: Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited.

By Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, Jie Tang

llms agents multimodal benchmarks

arXiv AI

Jul 21

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

arXiv:2507. 05515v3 Announce Type: replace Abstract: Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required.

By Haochen Huang, Yue Su, Xin Sun, Moonisa Ahsan, Mohammad Aliannejadi, Irene Viola, Zhaochun Ren, Chuang Yu, Aneta Lisowska, Artem Belopolsky, Koen Hindriks, Pablo Cesar, Junxiao Wang, Jiahuan Pei

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 21

Sparse Evidence Can Suffice: Agentic Evidence Seeking for Multimodal Video Misinformation Detection

arXiv:2607. 18080v1 Announce Type: cross Abstract: Multimodal video misinformation detection is commonly formulated as a holistic video-understanding task, where the entire video and its associated content are processed and judged in a single pass.

By Haochen Zhao, Yongxiu Xu, Xinkui Lin, Dong Xie, Jiarui Lu, Yuqi Qian, Yubin Wang, Hongbo Xu, Gaopeng Gou

agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 21

DeeperRadar: End-to-End MIMO Radar Design and Multi-Modal Fusion for Autonomous Vehicle Perception

arXiv:2607. 17351v1 Announce Type: new Abstract: DeeperRadar is a radar-centric, sensor-stack-conditioned framework that co-designs radar sensing and multi-modal 3D detection for autonomous mobility by learning a sparse acquisition pattern end-to-end with the fusion model.

By Eli Goldenshluger, Barak Pinkovich, Chaim Baskin

agents multimodal

arXiv AI

Jul 21

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

arXiv:2607. 18230v1 Announce Type: cross Abstract: Modern vision-language models (VLMs) have significantly improved image generation and editing capabilities, making pixel-level image tampering detection increasingly important yet challenging under cross-model and out-of-distribution shifts.

By Yi Tang, Xinyi Shang, Jiacheng Cui, Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tran Dinh Tien, Ahmed Elhagry, Salwa K. Al Khatib, Tianjun Yao, Yonina C. Eldar, Jing-Hao Xue, Hao Li, Salman Khan, Zhiqiang Shen

llms diffusion robotics multimodal benchmarks

arXiv AI

Jul 21

Neural Global Optimization via Iterative Refinement from Noisy Samples

arXiv:2604. 03614v2 Announce Type: replace-cross Abstract: Global optimization of black-box functions from noisy samples is a fundamental challenge in machine learning and scientific computing.

By Qusay Muzaffar, David Levin, Michael Werman

multimodal

arXiv AI

Jul 21

Data-Efficient Curation for Multimodal Reasoning under Fixed Training Protocols

arXiv:2601. 10922v2 Announce Type: replace Abstract: We study data curation for multimodal reasoning in a fixed-protocol fine-tuning regime, where the base model, optimizer, training schedule, and evaluation pipeline are held constant and the main degree of freedom is the training data.

By Yosub Shin, Michael Buriek, Boris Sobolev, Pavel Bushuyeu, Vikas Kumar, Haoyang Xu, Samuel Watson, Igor Molybog

fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

arXiv:2607. 13125v2 Announce Type: replace-cross Abstract: We introduce Boogu-Image-0.

By Guoxuan Chen, Chufeng Xiao, Haoran Yang, Siyue Xie, Binxiao Huang, Ming Zhang, Cheuk Him Chau, Xinyu Fu, Yingzhao Lian, Tom S. Y. Li, Jintao Lin, Bowen Dong, Zian Qian, Yuhao Liu, Yuxuan Hu, Weikang Shi, Bin Zou, Bowen Zheng, Haoxuan Che, Chang Chen, Yuyang He, Heyang Sun, Tianyu Huang, Chong Hou Choi, Cheng Gong, Han Shi, Haoli Bai, Xihui Liu, Hongsheng Li, Qifeng Chen, Chao Huang, Rui Liu, Chenyang Lei

llms agents diffusion multimodal benchmarks data-engineering

Get the daily briefing