Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,964 stories · RSS feed

arXiv AI
Jul 14

Can Argus Judge Them All? Comparing VLMs Across Domains

arXiv:2507. 01042v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly used in industry VLM applications such as retrieval systems, content generation platforms, and decision-support workflows, where model selection is commonly guided by benchmark rankings.

By Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali, Ebad Shabbir, Niharika Jain, Sarthak Jain, Jiechao Gao, Usman Naseem
arXiv AI
Jul 14

LightMem-Ego: Your AI Memory for Everyday Life

arXiv:2607. 11487v1 Announce Type: cross Abstract: Personal AI assistants on mobile and wearable devices continuously perceive users' daily lives through visual and audio streams.

By Yijun Chen, Boyi Xiao, Yixian Zhao, Haoting Xia, Buqiang Xu, Jizhan Fang, Yanya Li, Yaqi Zheng, Xuehai Wang, Zirui Xue, Liuxin Zhang, Hui Li, Ningyu Zhang
arXiv AI
Jul 14

Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

arXiv:2607. 09785v1 Announce Type: cross Abstract: Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams.

By Sergi Masip, Alicja Dobrzeniecka, Jonathan Swinnen, Joachim Collin, Bart{\l}omiej Twardowski, Szymon {\L}ukasik, Tinne Tuytelaars
arXiv AI
Jul 14

SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning

arXiv:2607. 10966v1 Announce Type: new Abstract: We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning.

By Mingyuan Wu, Jingcheng Yang, Shengyi Qian, Xudong Wang, Jize Jiang, Qifan Wang, Aashu Singh, Khoi Pham, Fei Liu, Zhaolun Su, Zhuokai Zhao, Klara Nahrstedt, Jianyu Wang, Hanchao Yu
arXiv Machine Learning
Jul 14

DynaFilter: Cloud-driven Dynamic Filtering for Satellite Edge Intelligence

arXiv:2607. 10098v1 Announce Type: cross Abstract: Modern satellite edge systems, including those performing remote sensing tasks such object detection and tracking, are characterized by severely limited bandwidth and intermittent connections, making continuous data transmission to the cloud impractical.

By Ziyang Zhang, Jie Liu, Luca Mottola
arXiv AI
Jul 14

MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models

arXiv:2607. 11594v1 Announce Type: new Abstract: Multi-scene navigation (clearing an objective in one bounded space and then crossing a portal into the next) is a defining feature of contemporary 3D games, but authoring it is laborious: every portal must have consistent endpoints on both sides, each interior must remain navigable once it is furnished, and the resulting connectivity must be kept consistent across many files.

By Tsz Hei Fan, Choi Wing Fung, Yuxuan Wan, Shuqing Li, Michael R. Lyu
arXiv Machine Learning
Jul 14

Imputation-free transformer learning enables robust Alzheimer's disease prediction and calibrated uncertainty quantification across heterogeneous clinical cohorts

arXiv:2607. 11656v1 Announce Type: cross Abstract: Accurate diagnostic classification and disease-severity prediction for Alzheimer's disease are hampered by the incompleteness and heterogeneity of real-world clinical data.

By Christelle Schneuwly Diaz, Narmina Baghirova, Duy-Thanh Vu, Duy-Cat Can, Gilles Allali, Philippe Ryvlin, Oliver Y. Ch\'en
arXiv AI
Jul 14

Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns

arXiv:2607. 11621v1 Announce Type: new Abstract: Aphasia following stroke commonly produces systematic naming errors with characteristic profiles, but whether general-purpose language models not designed for clinical simulation can reproduce these patterns remains untested.

By Yong Yang, Xiang Guan, Sophie Arheix-Parras, Saeed Ahmadi, Roger Newman-Norlund, Leonardo Bonilha, Christopher Rorden, Julius Fridriksson, Rutvik H. Desai, Srihari Nelakuditi
arXiv AI
Jul 14

The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning

arXiv:2607. 11436v1 Announce Type: new Abstract: Vision-language models increasingly succeed on multimodal reasoning benchmarks, yet their visual evidence often becomes unstable once it enters the language stack, weakening evidence-grounded reasoning.

By Wencheng Ye, Yi Bin, Yujuan Ding, Hongye Fang, Zheng Wang, Xing Xu, Jingkuan Song, Yun Zhang, Sirui Da, Heng Tao Shen
arXiv Machine Learning
Jul 14

From Direction to Magnitude: How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States

arXiv:2607. 09842v1 Announce Type: new Abstract: We investigate whether identity-specifying system prompts produce statistically distinguishable geometric fingerprints in the hidden-state trajectories of four open-weight transformer language models spanning four post-training regimes: no training (Gemma-4-E4B base), multimodal RLHF (Gemma-4-E4B-it), RL distillation (DeepSeek-R1-Distill-Qwen-7B), and SFT (Qwen2.

By Jorge A. Castillo, Marco Torres Y\'evenes, Juan Carlos Lanas
arXiv AI
Jul 14

NVAITC AI Scientist: A Governed End-to-End Research System -- A Hypertension GWAS Case Study

arXiv:2607. 11084v1 Announce Type: new Abstract: Agentic research systems are emerging as a new paradigm for coordinating scientific workflows beyond isolated model inference, code generation, or statistical analysis.

By Eddie Huang (NVIDIA AI Technology Center, NVIDIA Corporation), Ken Liao (NVIDIA AI Technology Center, NVIDIA Corporation), Iven Fu (NVIDIA AI Technology Center, NVIDIA Corporation), Yang-Hsien Lin (NVIDIA AI Technology Center, NVIDIA Corporation), Chao-Shun Zhan (NVIDIA AI Technology Center, NVIDIA Corporation), Andy Liao (NVIDIA AI Technology Center, NVIDIA Corporation), Virginia Chen (NVIDIA AI Technology Center, NVIDIA Corporation), Johnson Sun (NVIDIA AI Technology Center, NVIDIA Corporation), Pika Wang (NVIDIA AI Technology Center, NVIDIA Corporation), Richard Huang (NVIDIA AI Technology Center, NVIDIA Corporation), Jiun-Cheng Jiang (NVIDIA AI Technology Center, NVIDIA Corporation), Ting-Yuan Liu (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan, Master Program for Digital Health Innovation, China Medical University, Taichung, Taiwan), Hsing-Fang Lu (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan, Laboratory for Statistical and Translational Genetics, RIKEN Center for Integrative Medical Sciences, Yokohama, Japan), Ray Y. Lee (AI-Driven Genomic Medicine and Drug Discovery Lab, China Medical University Hospital, Taichung, Taiwan), Chi-Chou Liao (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan), Simon See (NVIDIA AI Technology Center, NVIDIA Corporation), Fuu-Jen Tsai (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan, Department of Medical Laboratory Science and Biotechnology, Asia University, Taichung, Taiwan)