Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,964 stories · RSS feed

arXiv AI

Jul 14

Quantum Circuit Vision: Cost-Aware Evaluation of Visual AI Agents for Quantum Code Generation

arXiv:2607. 10057v1 Announce Type: cross Abstract: Can AI agents visually comprehend quantum circuit diagrams and generate verified executable code--and at what cost?

By Dongping Liu, Aoyu Zhang, Luyao Zhang

llms agents multimodal benchmarks

arXiv AI

Jul 14

ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

arXiv:2607. 10180v1 Announce Type: cross Abstract: We introduce ActiveFly-Bench, the first benchmark to bridge cyberspace reasoning and physical-world interaction for UAV embodied perception.

By Weichen Zhang, Shiquan Yu, Yinan Zhu, Peizhi Tang, Shilong Ji, Zhiyuan Deng, Tianyi Lyu, Haoyang Wang, Xin Zeng, Chen Gao, Yong Li, Xinlei Chen

agents nlp robotics multimodal benchmarks

arXiv AI

Jul 14

LOGOS: A Living Logic for AI Agent Teams That Evolve With Humans

arXiv:2607. 10878v1 Announce Type: new Abstract: AI agents are evolving from answer engines into persistent teams that use tools, delegate work, learn from experience, and modify the artifacts that shape their future behavior.

By Yuma Ichikawa, Yamato Arai, Kosaku Kimura, Akira Sakai, Hiromichi Kobashi

agents multimodal

arXiv Machine Learning

Jul 14

Towards Real-World Wearable Motion Reconstruction

arXiv:2607. 09780v1 Announce Type: cross Abstract: The modern-day surge in popularity of wearable devices poses a fundamentally unique motion capture problem: reconstructing full-body movement from any set of sensing hardware worn at a given moment.

By Andrea Boscolo Camiletto, Rishabh Dabral, Eduardo Alvarado, Thabo Beeler, Marc Habermann, Christian Theobalt

multimodal

arXiv AI

Jul 14

PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning

arXiv:2607. 10190v1 Announce Type: cross Abstract: Video-language models (VLMs) have achieved remarkable performance on video understanding and visual question answering, yet they remain unreliable in reasoning about physical plausibility, where understanding object interactions, causal dynamics, and fundamental physical principles is essential.

By Wenyuan Wang, Lianyu Hu, Hao Wang, Yang Liu

llms rag nlp fine-tuning multimodal benchmarks

arXiv AI

Jul 14

Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data

arXiv:2607. 09876v1 Announce Type: cross Abstract: Automatically retrieving videos from large camera-trap datasets remains challenging.

By Valentin Gabeff, Baptiste Maquignaz, Jennifer Shan, Sepideh Mamooler, Gencer Sumbul, Blair Costelloe, Devis Tuia, Alexander Mathis

llms agents diffusion computer-vision multimodal benchmarks safety

arXiv Machine Learning

Jul 14

Error Aware Distribution Prediction for Lightweight Implicit Neural Representations

arXiv:2607. 10068v1 Announce Type: new Abstract: Implicit neural representations (INRs) offer compact encoding of volumes, but as lossy approximators, inevitably have prediction errors.

By Zhimin Li, Jake D. Balla, Joshua A. Levine

multimodal

arXiv AI

Jul 14

Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMs

arXiv:2607. 10666v1 Announce Type: cross Abstract: Deploying AI-based visual inspection in manufacturing is hard because requirements change often, new defect types appear, and large labeled datasets are rarely available.

By Shubham Rao

llms fine-tuning efficiency multimodal

arXiv Machine Learning

Jul 14

TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddings

arXiv:2607. 11007v1 Announce Type: new Abstract: Few-shot multimodal classification commonly attaches a lightweight head, such as $k$-nearest neighbors, logistic regression, or a linear SVM, to a frozen pretrained encoder.

By Jingxiang Zhang, Lujia Zhong, Zijie Zhu, Shuo Huang, Yuang Xu

rag multimodal

arXiv AI

Jul 14

NVAITC AI Scientist: A Governed End-to-End Research System -- A Hypertension GWAS Case Study

arXiv:2607. 11084v1 Announce Type: new Abstract: Agentic research systems are emerging as a new paradigm for coordinating scientific workflows beyond isolated model inference, code generation, or statistical analysis.

By Eddie Huang (NVIDIA AI Technology Center, NVIDIA Corporation), Ken Liao (NVIDIA AI Technology Center, NVIDIA Corporation), Iven Fu (NVIDIA AI Technology Center, NVIDIA Corporation), Yang-Hsien Lin (NVIDIA AI Technology Center, NVIDIA Corporation), Chao-Shun Zhan (NVIDIA AI Technology Center, NVIDIA Corporation), Andy Liao (NVIDIA AI Technology Center, NVIDIA Corporation), Virginia Chen (NVIDIA AI Technology Center, NVIDIA Corporation), Johnson Sun (NVIDIA AI Technology Center, NVIDIA Corporation), Pika Wang (NVIDIA AI Technology Center, NVIDIA Corporation), Richard Huang (NVIDIA AI Technology Center, NVIDIA Corporation), Jiun-Cheng Jiang (NVIDIA AI Technology Center, NVIDIA Corporation), Ting-Yuan Liu (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan, Master Program for Digital Health Innovation, China Medical University, Taichung, Taiwan), Hsing-Fang Lu (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan, Laboratory for Statistical and Translational Genetics, RIKEN Center for Integrative Medical Sciences, Yokohama, Japan), Ray Y. Lee (AI-Driven Genomic Medicine and Drug Discovery Lab, China Medical University Hospital, Taichung, Taiwan), Chi-Chou Liao (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan), Simon See (NVIDIA AI Technology Center, NVIDIA Corporation), Fuu-Jen Tsai (Department of Medical Research, China Medical University Hospital, Taichung, Taiwan, Department of Medical Laboratory Science and Biotechnology, Asia University, Taichung, Taiwan)

agents multimodal safety

arXiv AI

Jul 14

LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training Environments

arXiv:2607. 11839v1 Announce Type: cross Abstract: This paper presents a cascaded Low-Rank Adaptation (LoRA)-based multimodal fusion framework for action and activity recognition in healthcare-oriented training environments.

By Divya Mereddy, Jeevan Beedareddy

fine-tuning multimodal

arXiv Machine Learning

Jul 14

ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning

arXiv:2607. 09884v1 Announce Type: cross Abstract: Multimodal medical models often degrade when inputs are missing, a common scenario in real-world clinical workflows.

By Nusrat Binta Nizam, Fengbei Liu, Sunwoo Kwak, Minh Nguyen, Ruining Deng, Mert R. Sabuncu

multimodal

arXiv Machine Learning

Jul 14

On the modality gap and the contrastive loss in multi-modal representation learning

arXiv:2607. 10698v1 Announce Type: new Abstract: We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text embeddings remain misaligned despite being trained in a shared space.

By Fabian Mager, Hiba Nassar, Lars Kai Hansen

rag multimodal benchmarks

arXiv Machine Learning

Jul 14

DynaFilter: Cloud-driven Dynamic Filtering for Satellite Edge Intelligence

arXiv:2607. 10098v1 Announce Type: cross Abstract: Modern satellite edge systems, including those performing remote sensing tasks such object detection and tracking, are characterized by severely limited bandwidth and intermittent connections, making continuous data transmission to the cloud impractical.

By Ziyang Zhang, Jie Liu, Luca Mottola

computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Jul 14

TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

arXiv:2607. 10762v1 Announce Type: cross Abstract: Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding.

By Sutharsan Mahendran, Darshana Priyasad, Kaushik Roy, Tharindu Fernando, Sridha Sridharan, Clinton Fookes, Peyman Moghadam

llms rag computer-vision efficiency multimodal

arXiv AI

Jul 14

Technical Report on the CVPR 2026@AdvML Workshop Challenge

arXiv:2607. 11560v1 Announce Type: cross Abstract: Vision-language agents (VLAs) are increasingly used to interpret complex driving scenes and support safety-critical reasoning.

By Tianyuan Zhang, Zonglei Jing, Jiangfan Liu, Ligong Zhang, Ke Ma, Chengzhi Sun, Xiaohai Xu, Zhirui Zhang, Qianqian Xu, Qingming Huang, Hanyu Fang, Junhua Liu, Zheng Wang, Xiaoliang Liu, Yuanbo Li, Shuai Gui, Bin Wang, Menghe Zheng, Jing Nie, Hanyang Meng, Zeyang Zhang, Xiang Zhang, Yongxuan Zhu, Rui Ding, Hainan Li, Yongkang Zhang, Zhilei Zhu, Xianglong Kong, Jin Hu, Zonghao Ying, Yisong Xiao, Lei Chen, Haotong Qin, Jiakai Wang, Aishan Liu, Ruikai Li, Julia Karbing, Yinpeng Dong, Zhenfei Yin, Shao Jing, Xia Hu, Jingyi Xu, Juntao Dai, Xinyun Chen, Vishal M. Patel, Xianglong Liu, Dawn Song, Alan Yuille, Philip H. S. Torr, Dacheng Tao

agents nlp multimodal benchmarks safety

arXiv AI

Jul 14

Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion

arXiv:2607. 11004v1 Announce Type: cross Abstract: We present a manipulation planning system based on affordance recognition and action effect prediction.

By Solvi Arnold, Rin Karashima, Tadashi Adachi, Takafumi Mochizuki, Kimitoshi Yamazaki

robotics multimodal

arXiv AI

Jul 14

Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

arXiv:2607. 10991v1 Announce Type: cross Abstract: As mobile robots become more integrated into everyday human environments, social robot navigation is becoming essential for ensuring human comfort, safety, and trust.

By Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Cretides, Marie Samson, Mahdi Khoramshahi, Mohamed Chetouani

llms reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jul 14

Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

arXiv:2607. 10358v1 Announce Type: cross Abstract: Foundation models are increasingly used as image feature extractors for mammography, but their robustness under external domain shift remains unclear.

By Giang Nguyen, Raghav Mehta, Emma A. M. Stanley, Tian Xia, Thi Hao Nguyen, Hieu Pham, Ben Glocker

llms multimodal benchmarks

arXiv Machine Learning

Jul 14

LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models

arXiv:2607. 11257v1 Announce Type: cross Abstract: Pathology Foundation Models (PFMs) offer powerful Whole Slide Image (WSI) representations but suffer from massive computational costs.

By Gangsu Kim, Won-Ki Jeong

nlp efficiency multimodal safety

Get the daily briefing