Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,869 stories · RSS feed

arXiv AI

Jul 29

DynaBridge: Dynamic Summary-Guided Cross-Task Multimodal Fusion for DASS-Structured Mental Health Assessment

arXiv:2607. 25679v1 Announce Type: cross Abstract: Multimodal behavioral analysis offers a scalable approach to assessing depression, anxiety, and stress, yet generic fusion models often ignore the psychometric structure of questionnaire labels.

By Shiyu Teng, Haichen Yu, Jiaqing Liu, Hao Sun, Yu Song, Shurong Chai, Ruibo Hou, Lanfen Lin, Yen-Wei Chen

llms multimodal

arXiv AI

Jul 29

Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs

arXiv:2607. 22205v2 Announce Type: replace-cross Abstract: Remote sensing multimodal large language models (RS-MLLMs) have improved general aerial-image understanding.

By Yuheng Zong, Minghua Wang, Xin Zhao, Zhi-Hui Zhan, Antonio Plaza, Jon Atli Benediktsson

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 29

InDex: Empowering VLA Models with Intent-Conditioned Arm-Hand Coordination for Dexterous Manipulation

arXiv:2606. 12109v2 Announce Type: replace-cross Abstract: Pre-trained Vision-Language-Action (VLA) models provide useful semantic and spatial priors, yet their parallel-gripper action interfaces do not specify how those priors should be realized by a dexterous hand.

By Chuanke Pang, Junyi Huang, Zhijun Zhao, Yaobing Wang, Kun Xu, Xilun Ding

diffusion robotics multimodal

arXiv AI

Jul 29

RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

arXiv:2512. 06276v3 Announce Type: replace-cross Abstract: Referring Expression Comprehension (REC) is a vision-language task that localizes a specific image region based on a textual description.

By Tianyi Gao, Hao Li, Han Fang, Xin Wei, Xiaodong Dong, Hongbo Sun, Ye Yuan, Zhongjiang He, Jinglin Xu, Jingmin Xin, Hao Sun

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jul 29

RankFormer: A Propose-then-Select Transformer for Multi-Agent Multimodal Trajectory Prediction

arXiv:2604. 07126v2 Announce Type: replace-cross Abstract: Predicting vehicle trajectories plays an important role in autonomous driving, transportation safety analysis, traffic operations, etc.

By Diyi Liu, Zihan Niu, Tu Xu, Xingchen Zhang, Lishan Sun

llms agents robotics multimodal safety

arXiv AI

Jul 29

Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphs

arXiv:2607. 25959v1 Announce Type: cross Abstract: Wikipedia and Wikidata are widely used for information access, LLM pre-training, and retrieval-augmented generation.

By Fanfu Wei, Thibault Ehrhart, Rapha\"el Troncy

llms rag multimodal benchmarks

arXiv AI

Jul 29

Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

arXiv:2505. 14366v2 Announce Type: replace Abstract: We present a conceptual framework for training Vision-Language Models (VLMs) to perform Visual Perspective Taking (VPT), a core capability for embodied cognition essential for Human-Robot Interaction (HRI).

By Joel Currie, Gioele Migno, Enrico Piacenti, Maria Elena Giannaccini, Patric Bach, Davide De Tommaso, Agnieszka Wykowska

llms robotics multimodal

arXiv AI

Jul 29

HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document

arXiv:2607. 24861v1 Announce Type: cross Abstract: Question answering (QA) over complex documents requires models to retrieve and integrate evidence distributed across distant document regions and modalities.

By Xin He, Yili Wang, Wenqi Fan, Qing Li, Qinggang Zhang, Yi Chang, Xin Wang

rag nlp multimodal

arXiv AI

Jul 29

From Cellular Responses to Pharmacological Domains: Multimodal Zero-Shot Drug Representation Learning

arXiv:2607. 25322v1 Announce Type: new Abstract: Multimodal drug discovery enables drug representation learning beyond chemical structure by incorporating cellular responses such as gene expression and cell morphology.

By Jintao Huang, Lu Leng, Ziyuan Yang

multimodal safety

arXiv AI

Jul 29

CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer

arXiv:2607. 26023v1 Announce Type: new Abstract: Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks.

By Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, Dongxiao He

llms fine-tuning multimodal

arXiv AI

Jul 29

VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents

arXiv:2607. 24748v1 Announce Type: cross Abstract: Visually-rich documents such as reports, slides, and manuals often distribute the evidence needed to answer a question across multiple pages, mixing text with layout cues, tables, charts, and figures.

By Seonok Kim

rag agents nlp multimodal benchmarks

arXiv AI

Jul 29

CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

arXiv:2607. 25294v1 Announce Type: cross Abstract: Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge.

By Lai Wei, Chengqi Li, Jiapeng Li, Ruina Hu, Yue Wang, Weiran Huang

nlp multimodal benchmarks

arXiv AI

Jul 29

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

arXiv:2607. 25948v1 Announce Type: cross Abstract: Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy.

By Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, Fran\c{c}ois Fleuret, Chuan Li, Amir Zadeh, Serge Belongie, Afshin Dehghan, Jesse Allardice, David Mizrahi, O\u{g}uzhan Fatih Kar, Roman Bachmann, Amir Zamir

llms diffusion multimodal benchmarks

arXiv AI

Jul 29

I2VShield: An Efficient Proactive Defense Framework against DiT-based Image-to-Video Models

arXiv:2607. 25522v1 Announce Type: cross Abstract: The rapid advancement of video generation models has led to the increasing misuse of image-to-video (I2V) models.

By Yimao Guo, Zuomin Qu, Wei Lu

llms diffusion multimodal safety

arXiv AI

Jul 29

AnnoBench: A Benchmark for Visualization Annotation Generation

arXiv:2607. 25911v1 Announce Type: cross Abstract: Annotation is among the most demanding visualization tasks to automate, as it simultaneously requires correctly navigating visual, semantic, and stylistic constraints.

By Md Rahat-uz-Zaman, Md Dilshadur Rahman, Andrew McNutt, Paul Rosen

multimodal benchmarks

arXiv AI

Jul 29

Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming

arXiv:2607. 25047v1 Announce Type: cross Abstract: Extended Reality (XR) is increasingly used in human-robot interaction to communicate robot intent, planned motion, reachability, and state.

By Jens Grubert, John Dudley, Eyal Ofek, Per Ola Kristensson

robotics multimodal

arXiv AI

Jul 29

ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop

arXiv:2607. 24770v1 Announce Type: new Abstract: Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions.

By Azizul Zahid, Subrata Biswas, Bashima Islam, Sai Swaminathan

llms agents nlp multimodal safety

arXiv AI

Jul 29

OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis

arXiv:2607. 25108v1 Announce Type: cross Abstract: Biomedical image analysis spans diverse modalities and tasks, yet real-world deployment is hindered by severe distribution shifts across scanners, protocols, and patient populations.

By Zihan Li, Feiyang Liu, Dandan Shan, Ruibo Wang, Qingqi Hong

agents computer-vision fine-tuning multimodal benchmarks safety

arXiv AI

Jul 29

Less is More: Modality-Decoupling for General AIGC Audio-Video Detection

arXiv:2607. 25543v1 Announce Type: cross Abstract: Generative AI has rapidly expanded audio-visual forgery beyond human-centric deepfakes into general scenes.

By Jielun Peng, Yabin Wang, Yaqi Li, Jincheng Liu, Xiaopeng Hong, Athanasios V. Vasilakos

multimodal

arXiv AI

Jul 29

$\pi\mathbf{R}^2$: Reactive Real-time Flow Policies

arXiv:2607. 26055v1 Announce Type: cross Abstract: Generalist manipulation policies increasingly take the form of action-chunking flow policies built on large pretrained backbones.

By Sungjae Park, Shubham Tulsiani

rag diffusion robotics multimodal

Get the daily briefing