Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,303 stories · RSS feed

arXiv AI

Jul 14

Large Multimodal Model-Based Environment-Aware Mobility Management

arXiv:2607. 09795v1 Announce Type: cross Abstract: Recently, large language models (LLMs) have been successfully adopted in various fields, including wireless communications, robotics, and autonomous vehicles, owing to their outstanding adaptability and reasoning abilities.

By Seokhyun Jeong, Sangmok Shin, Seungnyun Kim, Jiao Wu, Byonghyo Shim

llms agents robotics multimodal

arXiv AI

Jul 14

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

arXiv:2604. 22823v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities.

By Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song, Qimeng Zhang, Haifeng Chen, Changsheng Xu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 14

A Multimodal Dataset for Large Language Model Applications in the Energy Domain

arXiv:2607. 11459v1 Announce Type: cross Abstract: This paper presents the mAIEnergy dataset, an open-access, multimodal corpus developed to support Large Language Model (LLM) applications in the energy sector.

By Costas Mylonas, Magda Foti

llms rag multimodal

arXiv AI

Jul 14

MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models

arXiv:2607. 11594v1 Announce Type: new Abstract: Multi-scene navigation (clearing an objective in one bounded space and then crossing a portal into the next) is a defining feature of contemporary 3D games, but authoring it is laborious: every portal must have consistent endpoints on both sides, each interior must remain navigable once it is furnished, and the resulting connectivity must be kept consistent across many files.

By Tsz Hei Fan, Choi Wing Fung, Yuxuan Wan, Shuqing Li, Michael R. Lyu

llms agents multimodal benchmarks

arXiv AI

Jul 14

Technical Report on the CVPR 2026@AdvML Workshop Challenge

arXiv:2607. 11560v1 Announce Type: cross Abstract: Vision-language agents (VLAs) are increasingly used to interpret complex driving scenes and support safety-critical reasoning.

By Tianyuan Zhang, Zonglei Jing, Jiangfan Liu, Ligong Zhang, Ke Ma, Chengzhi Sun, Xiaohai Xu, Zhirui Zhang, Qianqian Xu, Qingming Huang, Hanyu Fang, Junhua Liu, Zheng Wang, Xiaoliang Liu, Yuanbo Li, Shuai Gui, Bin Wang, Menghe Zheng, Jing Nie, Hanyang Meng, Zeyang Zhang, Xiang Zhang, Yongxuan Zhu, Rui Ding, Hainan Li, Yongkang Zhang, Zhilei Zhu, Xianglong Kong, Jin Hu, Zonghao Ying, Yisong Xiao, Lei Chen, Haotong Qin, Jiakai Wang, Aishan Liu, Ruikai Li, Julia Karbing, Yinpeng Dong, Zhenfei Yin, Shao Jing, Xia Hu, Jingyi Xu, Juntao Dai, Xinyun Chen, Vishal M. Patel, Xianglong Liu, Dawn Song, Alan Yuille, Philip H. S. Torr, Dacheng Tao

agents nlp multimodal benchmarks safety

arXiv AI

Jul 14

Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification

arXiv:2607. 10706v1 Announce Type: cross Abstract: The action space poses a major challenge in robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions.

By Haojie Huang, Zhang Ye, Linfeng Zhao, Boce Hu, Mingxi Jia, Yu Qi, Ahmed Agha, Dian Wang, Robert Platt, Robin Walters

llms diffusion robotics multimodal

arXiv AI

Jul 14

Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models

arXiv:2603. 22042v3 Announce Type: replace-cross Abstract: While Vision-Language Models (VLMs) have achieved remarkable performance, their Euclidean embeddings remain limited in capturing hierarchical relationships such as part-to-whole or parent-child structures, and often face challenges in multi-object compositional scenarios.

By Hayeon Kim, Ji Ha Jang, Junghun James Kim, Se Young Chun

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jul 14

TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation

arXiv:2607. 10762v1 Announce Type: cross Abstract: Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding.

By Sutharsan Mahendran, Darshana Priyasad, Kaushik Roy, Tharindu Fernando, Sridha Sridharan, Clinton Fookes, Peyman Moghadam

llms rag computer-vision efficiency multimodal

arXiv AI

Jul 14

VINE: Taming Generative Control Policies for Reinforcement Learning

arXiv:2607. 10369v1 Announce Type: cross Abstract: Flow-matching policies have emerged as an effective policy parameterization for robot learning.

By Rushuai Yang, Zhuo Han, Houlin Li, Hecheng Wang, Zhichao Wu, Rui Zhang, Zhaowei Zhang, Zihong Chen, Xiaohan Yan, Chiming Liu, Yi Chen, Wei Shan, Maoqing Yao

reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jul 14

3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

arXiv:2607. 10826v1 Announce Type: cross Abstract: Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow.

By Zhenyu Zhao, Nanshan Jia, Jihyeon Je, Yifu Tang, Alvin Chan, Michael Spedden, Michael V. Palleschi, Sui Huang, Jingshen Wang, Zeyu Zheng

llms multimodal benchmarks

arXiv Machine Learning

Jul 14

Memory Savings at What Cost? A Study of Alternatives to Backpropagation

arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.

By Kunjal Panchal, Sunav Choudhary, Yuriy Brun, Hui Guan

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 14

EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation

arXiv:2607. 10165v1 Announce Type: cross Abstract: Emotion-aware artistic image generation requires an image to match the input prompt, follow the specified artistic style, and convey the target emotion.

By Dexiang Hong, Yijie Guo, Weidong Chen, Xinyan Liu, Zixuan Zou, Zhendong Mao, Yongdong Zhang

llms diffusion fine-tuning multimodal safety

arXiv AI

Jul 14

Source-Lifted Flow Matching for Intervenable Multimodal Imitation

arXiv:2607. 10206v1 Announce Type: cross Abstract: Flow-matching policies are promising for imitation learning because they model complex multimodal action distributions.

By He Zhang, Ying Sun, Pengteng Li, Ziyang Chen, Yiren Zhao, Ziyang Rao, Weiyu Guo, Yandong Guo, Hui Xiong

diffusion robotics multimodal benchmarks

arXiv AI

Jul 14

MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

arXiv:2604. 04969v2 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning.

By Sijun Dai, Qiang Huang, Xiaoxing You, Jun Yu

llms rag multimodal benchmarks safety

arXiv AI

Jul 14

Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data

arXiv:2607. 09876v1 Announce Type: cross Abstract: Automatically retrieving videos from large camera-trap datasets remains challenging.

By Valentin Gabeff, Baptiste Maquignaz, Jennifer Shan, Sepideh Mamooler, Gencer Sumbul, Blair Costelloe, Devis Tuia, Alexander Mathis

llms agents diffusion computer-vision multimodal benchmarks safety

arXiv Machine Learning

Jul 14

DynaFilter: Cloud-driven Dynamic Filtering for Satellite Edge Intelligence

arXiv:2607. 10098v1 Announce Type: cross Abstract: Modern satellite edge systems, including those performing remote sensing tasks such object detection and tracking, are characterized by severely limited bandwidth and intermittent connections, making continuous data transmission to the cloud impractical.

By Ziyang Zhang, Jie Liu, Luca Mottola

computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Jul 14

Artificial Intelligence Across the Cardiac Amyloidosis Diagnostic Pathway: From Single-Modality Detection to Multimodal Clinical Integration

arXiv:2607. 09948v1 Announce Type: cross Abstract: Cardiac amyloidosis (CA) is increasingly recognized but remains substantially underdiagnosed, because its clinical and imaging phenotype overlaps with more common cardiomyopathies.

By Diana Shadibaeva, Rochak Dhakal, Kui Zhang, Xiaofeng Yang, Saurabh Malhotra, Weihua Zhou

multimodal

arXiv Machine Learning

Jul 14

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

arXiv:2607. 09982v1 Announce Type: new Abstract: Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance.

By Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

multimodal safety

arXiv Machine Learning

Jul 14

ShapKO: Shapley-Adaptive Modality Knockout for Robust Multimodal Learning

arXiv:2607. 09884v1 Announce Type: cross Abstract: Multimodal medical models often degrade when inputs are missing, a common scenario in real-world clinical workflows.

By Nusrat Binta Nizam, Fengbei Liu, Sunwoo Kwak, Minh Nguyen, Ruining Deng, Mert R. Sabuncu

multimodal

arXiv AI

Jul 14

See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

arXiv:2607. 11498v1 Announce Type: cross Abstract: Vision-language-action (VLA) models predict robot actions from visual observations and language instructions.

By Byungkun Lee, Dongyoon Hwang, Dongjin Kim, Hojoon Lee, Minho Park, Jaegul Choo

robotics multimodal

Get the daily briefing