Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,029 stories · RSS feed

arXiv AI

Jul 13

Event Stream based Multi-Modal Video Anomaly Detection: A Benchmark Dataset and Algorithms

arXiv:2607. 09114v1 Announce Type: cross Abstract: Video anomaly detection (VAD) is critical for automated surveillance but remains fragile under challenging conditions such as illumination variations, fast motion, and complex backgrounds when relying solely on visible light videos.

By Peipei Zhu, Yueqing Niu, Lin Zhu, Guanchong Niu, Yang Yu, Zheng Li

rag multimodal benchmarks

arXiv AI

Jul 13

Omni-Sleep: A Sleep Foundation Model via Hierarchical Contrastive Learning of CNS-ANS Dynamics

arXiv:2607. 07720v2 Announce Type: replace-cross Abstract: Sleep physiology arises from the coordinated dynamics of the central nervous system (CNS) and autonomic nervous system (ANS), as reflected by multimodal polysomnography signals including EEG, EOG, EMG, ECG, and respiration.

By Zhoujie Hou, Song Wang, Kexin Lou, Mo Wang, Chen Wei, Quanying Liu

multimodal benchmarks

arXiv Machine Learning

Jul 13

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

arXiv:2607. 09450v1 Announce Type: cross Abstract: Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations.

By Xingyu Zhu, Huanshen Wu, Shuo Wang, Beier Zhu, Jiannan Ge, Jiaheng Zhang, Long Chen

llms multimodal safety

arXiv Machine Learning

Jul 13

Contrastive Learning on Multimodal Analysis of Electronic Health Records

arXiv:2403. 14926v3 Announce Type: replace-cross Abstract: Electronic health record (EHR) systems capture a wealth of multimodal clinical data, encompassing both structured clinical codes and unstructured clinical notes.

By Tianxi Cai, Feiqing Huang, Ryumei Nakada, Linjun Zhang, Doudou Zhou

rag multimodal safety

arXiv Machine Learning

Jul 13

The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs

arXiv:2607. 09544v1 Announce Type: cross Abstract: Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting.

By Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh, Nikolai Rozanov, Kentaro Inui

llms multimodal

arXiv AI

Jul 13

4DR360: State Reasoning for Joint 3D Detection and Occupancy Prediction in 4D Radar-Camera Full-Scene Perception

arXiv:2607. 09629v1 Announce Type: cross Abstract: Reliable autonomous driving requires full-scene perception that couples foreground objects with dense semantic layout.

By Xiaokai Bai, Lianqing Zheng, Runwei Guan, Songkai Wang, Siyuan Cao, Hui-liang Shen

agents robotics multimodal benchmarks

arXiv AI

Jul 13

SVF-CR: Synchronized Visual-Facial Cross-Refinement for Multimodal Ambivalence and Hesitancy Recognition

arXiv:2607. 09417v1 Announce Type: cross Abstract: Ambivalence and hesitancy are subtle behavioral states that are expressed through a combination of verbal content, facial behavior, visual context, and acoustic cues.

By Hyein Park, Namho Kim, Junhwa Kim

multimodal

arXiv AI

Jul 13

ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models

arXiv:2607. 09134v1 Announce Type: cross Abstract: Representation alignment (REPA) has been investigated to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity.

By Sang-Hoon Lee, Ha-Yeong Choi

llms diffusion multimodal safety

arXiv AI

Jul 13

M4V: Multimodal Mamba for Efficient Text-to-Video Generation

arXiv:2506. 10915v2 Announce Type: replace-cross Abstract: Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators.

By Jiancheng Huang, Gengwei Zhang, Zequn Jie, Siyu Jiao, Yinlong Qian, Ling Chen, Yunchao Wei, Lin Ma

llms diffusion multimodal benchmarks

arXiv AI

Jul 13

ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts

arXiv:2607. 09526v1 Announce Type: cross Abstract: Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones.

By Jiawen Li, Tian Guan, Huijuan Shi, Xitong Ling, Mingxi Fu, Anjia Han, Chao He, Yonghong He

efficiency multimodal

arXiv AI

Jul 13

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

arXiv:2607. 09438v1 Announce Type: cross Abstract: Test-time scaling (TTS) reliably improves reasoning in large language models, but whether it transfers to small open vision-language models remains unclear.

By Spiros Baxevanakis, Peng-Jian Yang

llms multimodal benchmarks

arXiv Machine Learning

Jul 13

Variable-Length Generative Protein Design via Generalized Poisson Flow

arXiv:2607. 09039v1 Announce Type: new Abstract: The ability to generate variable-length proteins is crucial in protein design, where the optimal length is often unknown and tightly coupled to designability.

By Chaoran Cheng, Zhanghan Ni, Yanru Qu, Yuxin Chen, Ruihan Guo, Jiajun Fan, Ge Liu

diffusion multimodal

arXiv Machine Learning

Jul 13

Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

arXiv:2607. 08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference.

By Dominick Reilly, Qiyu Wu, Hiromi Wakaki, Srijan Das, Yuki Mistufuji

llms multimodal safety

arXiv AI

Jul 13

Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

arXiv:2605. 28215v2 Announce Type: replace Abstract: In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labelled examples.

By Carmen Quiles-Ram\'irez, Leticia L. Rodr\'iguez, Nicol\'as Martorell, Natalia D\'iaz-Rodr\'iguez

llms multimodal benchmarks

arXiv AI

Jul 13

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

arXiv:2607. 08964v1 Announce Type: new Abstract: AI agents have become capable of autonomously completing short, well-specified tasks.

By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, LeoweiLiang

agents multimodal benchmarks

arXiv AI

Jul 13

SAGEAgent: A Self-Evolving Agent for Cost-Aware Modality Acquisition in Multimodal Survival Prediction

arXiv:2607. 09521v1 Announce Type: new Abstract: Does every cancer patient truly need a complete diagnostic workup for accurate survival prediction?

By Chongyu Qu, Can Cui, Zhengyi Lu, Junchao Zhu, Tianyuan Yao, Junlin Guo, Juming Xiong, Yanfan Zhu, Yuechen Yang, Bennett A. Landman, Yuankai Huo

llms agents multimodal

arXiv AI

Jul 13

Video Generation Models are General-Purpose Vision Learners

arXiv:2607. 09024v1 Announce Type: cross Abstract: Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models.

By Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu

diffusion computer-vision nlp robotics multimodal benchmarks safety

arXiv AI

Jul 13

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

arXiv:2607. 09654v1 Announce Type: cross Abstract: Vision language models (VLMs) have made remarkable progress in visual reasoning during the last decade.

By Shravan Murlidaran, Miguel P. Eckstein

llms computer-vision multimodal benchmarks safety

arXiv AI

Jul 13

ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AI

arXiv:2607. 09649v1 Announce Type: new Abstract: Concept-based explainable artificial intelligence (AI) can make model reasoning more human-understandable, but concept-level outputs are not automatically trustworthy.

By Mohadeseh Mollapour, Koorosh Aslansefat, Zeinab Dehghani, Bhupesh Kumar Mishra, Tejal Shah, Zhibao Mian

multimodal

arXiv AI

Jul 13

Task-Specific Multimodal Question Answering Agents via Confidence Calibration and Incremental Reasoning for QANTA 2026

arXiv:2607. 09623v1 Announce Type: cross Abstract: We present our submission to the QANTA 2026 shared challenge at the ICML 2026 Workshop on Efficient Multimodal Question Answering (EMM-QA).

By Nirjhar Das, Md. Al-Mamun Provath

llms agents nlp multimodal benchmarks

Get the daily briefing