Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,891 stories · RSS feed

arXiv AI

Aug 6

The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering

arXiv:2608. 04589v1 Announce Type: cross Abstract: EgoCross is a cross-domain egocentric video question answering benchmark designed to evaluate whether multimodal large language models can generalize beyond common daily-life scenarios.

By Yuqian Fu, Tianwen Qian, Yanjun Li, Yu Li, Kunyu Peng, Xu Zheng, Yongqin Xian, Alessio Tonioni, Yanwei Fu, Xiaoling Wang, Danda Paudel, Federico Tombari, Luc Van Gool, Leyi Wu, Yifan Zhao, Jinjie Zhang, Yinchuan Li, Yingcong Chen, Zixu Li, Zhiwei Chen, Zhiheng Fu, Wenbo Wang, Yupeng Hu, Weili Guan, Liqiang Nie, Takuya Murakawa, Toru Tamaki, Yi Wen, Zhenglin Du, Zhengyang Li, Lingling Li, Licheng Jiao, Wenping Ma

llms nlp multimodal benchmarks

arXiv AI

Aug 6

GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAs

arXiv:2608. 04510v1 Announce Type: cross Abstract: Diffusion-based vision-language-action (VLA) policies can generate plausible actions even when their predictions are weakly grounded in the visual and language evidence defining the task.

By Suhas Hegde, Jitendra Yasaswi Bharadwaj Katta

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Aug 6

Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference

arXiv:2608. 04428v1 Announce Type: cross Abstract: Vision-language-action (VLA) models have emerged as a key component in embodied AI.

By Zheng Liu, Zeyu Guo, Zihan Liu, Anbang Wu, Han Zhao, Fangxin Liu, Zhezhi He, Yinhe Han, Jingwen Leng, Minyi Guo, Yiming Gan, Yu Feng

diffusion robotics efficiency multimodal

arXiv AI

Aug 6

AI-driven Multimodal Representation Learning for Latent Mediation Structure Discovery of Socioeconomic Disadvantage, Psychosocial Factors, and Cardiometabolic Multimorbidity: Insights from the All of Us Research Program

arXiv:2608. 04016v1 Announce Type: cross Abstract: Social disadvantage is associated with multimorbidity, but the pathways linking social conditions to disease burden remain poorly understood.

By Cong Cao, Shuangge Ma

diffusion multimodal

arXiv AI

Aug 6

OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

arXiv:2608. 05013v1 Announce Type: cross Abstract: LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life.

By Jingsheng Zheng, Xinyuan Fang, Jintian Zhang, Zhengke Gui, Huajun Chen, Ningyu Zhang

llms agents multimodal

arXiv AI

Aug 6

TRNet: Topography-Guided Frequency Rectification and Structure-Aware Decoding for Multimodal Paddy Rice Segmentation

arXiv:2608. 04154v1 Announce Type: cross Abstract: Mapping paddy rice from very-high-resolution imagery in mountainous and hilly regions is difficult because terrain alters optical appearance and increases confusion with visually similar vegetation.

By Kaiwen Xiao, Chunlong Fu, Liping Zheng, Yanfeng Su

computer-vision multimodal benchmarks

arXiv AI

Aug 6

Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding

arXiv:2608. 04054v1 Announce Type: cross Abstract: Multimodal intent recognition requires understanding not only what textual, acoustic, and visual signals share, but also how they disagree.

By Mohnish Raj, Suraj Kumar, Soumi Chattopadhayay, Chandranath Adak, Ayan Dutta

multimodal benchmarks safety

arXiv Machine Learning

Aug 6

One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP

arXiv:2505. 19840v3 Announce Type: replace-cross Abstract: Deep Neural Networks (DNNs) have achieved widespread success yet remain prone to adversarial attacks.

By Binyan Xu, Xilin Dai, Di Tang, Kehuan Zhang

llms multimodal safety

arXiv AI

Aug 6

RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation

arXiv:2510. 17640v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have shown strong manipulation capability when trained with large-scale imitation learning datasets.

By Yuquan Xue, Guanxing Lu, Zhenyu Wu, Chuanrui Zhang, Bofang Jia, Zhengyi Gu, Ziwei Wang

robotics multimodal benchmarks

arXiv AI

Aug 6

C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning

arXiv:2608. 04013v1 Announce Type: cross Abstract: Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs.

By Yuntao Shou, Tao Meng, Wei Ai, Keqin Li

multimodal benchmarks

arXiv AI

Aug 6

Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Study

arXiv:2604. 07558v3 Announce Type: replace-cross Abstract: Digital mental health (DMH) tools have extensively explored personalization of interventions to users' needs and contexts.

By Ananya Bhattacharjee, Michael Liut, Matthew J\"orke, Diyi Yang, Emma Brunskill

llms multimodal

arXiv AI

Aug 6

MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding

arXiv:2604. 00513v3 Announce Type: replace-cross Abstract: With the rapid growth of e-commerce, exploring general representations rather than task-specific ones has attracted increasing attention.

By Junxian Wu, Chenghan Fu, Zhanheng Nie, Daoze Zhang, Bowen Wan, Wanxian Guan, Chuan Yu, Jian Xu, Bo Zheng

llms rag reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Aug 6

Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

arXiv:2604. 14888v3 Announce Type: replace-cross Abstract: Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear.

By Danae S\'anchez Villegas, Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott

llms multimodal safety

arXiv AI

Aug 6

DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models

arXiv:2511. 15669v3 Announce Type: replace-cross Abstract: Does Chain-of-Thought (CoT) reasoning genuinely improve Vision Language Action (VLA) models, or does it merely add overhead?

By Cheng Yin, Yankai Lin, Wang Xu, Sikyuen Tam, Xiangrui Zeng, Zhiyuan Liu, Zhouping Yin

reinforcement-learning robotics fine-tuning multimodal safety

arXiv AI

Aug 6

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

arXiv:2608. 04726v1 Announce Type: new Abstract: Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels.

By Yongxin Wang, Ruizhe Zhou, Yueling Tang, Yingying Zhu, Xuemin Zhao, Xiaojun Chang, Xiaodan Liang

llms computer-vision multimodal benchmarks

arXiv AI

Aug 6

SleepVLM: A Rule-Grounded Vision-Language Model for Auditable Sleep Staging

arXiv:2603. 26738v4 Announce Type: replace-cross Abstract: Sleep staging is essential for sleep assessment and disorder diagnosis.

By Guifeng Deng, Pan Wang, Mengfan Niu, Jiquan Wang, Shuying Rao, Junyi Xie, Xi'ang Chen, Sha Zhao, Gang Pan, Wanjun Guo, Tao Li, Haiteng Jiang

llms fine-tuning multimodal benchmarks safety

arXiv AI

Aug 6

Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

arXiv:2511. 19418v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.

By Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, XuDong Wang

llms computer-vision multimodal benchmarks safety

arXiv Machine Learning

Aug 6

Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Flow-matching

arXiv:2608. 05103v1 Announce Type: new Abstract: Data assimilation (DA) uses Bayesian inference to update the state of a numerical forecast model with observed data.

By Dibyajyoti Chakraborty, Romit Maulik

multimodal benchmarks

arXiv AI

Aug 6

PURPOSE: Poisoning Conflict Resolution in RAG via Proxy-Fact-Grounded Updates

arXiv:2608. 04756v1 Announce Type: cross Abstract: In Retrieval-Augmented Generation (RAG), post-retrieval conflict resolution arbitrates among noisy or contradictory retrieved passages.

By Zijian Wang, Yubo Zhu, Muzhi Dong, Yanjun Lou, Yisheng Li, ZiLiang Zhang, Wei Tong, Yuan Zhang, Jingyu Hua, Sheng Zhong

rag multimodal benchmarks

arXiv AI

Aug 6

Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering

arXiv:2608. 04124v1 Announce Type: cross Abstract: Video question answering requires models to ground language queries in visual evidence and, when necessary, reason over that evidence across time.

By Haotian Xia, Zilin Xiao, Junbo Zou, Vicente Ordonez, Hanjie Chen

llms diffusion nlp reinforcement-learning multimodal benchmarks