Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,972 stories · RSS feed

arXiv Machine Learning

Jul 1

Structured SIR: Efficient and Expressive Importance-Weighted Inference for High-Dimensional Image Registration

arXiv:2603. 17415v2 Announce Type: replace-cross Abstract: Image registration is an ill-posed dense vision task, where multiple solutions achieve similar loss values, motivating probabilistic inference.

By Ivor J. A. Simpson, Neill D. F. Campbell

multimodal

arXiv Machine Learning

Jul 1

InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context

arXiv:2603. 05353v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) for long-context question answering is bottlenecked by inference-time prefilling over large retrieved contexts.

By Xin Teng, Canyu Zhang, Shaoyi Zheng, Danyang Zhuo, Tianyi Zhou, Shenji Wan

llms rag nlp multimodal benchmarks

arXiv AI

Jul 1

Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents

arXiv:2606. 31270v1 Announce Type: cross Abstract: Computer-use agents, which leverage multimodal large language models (MLLMs) to operate computers and complete tasks, have attracted significant attention for their utility and versatility.

By Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt, Serena Yeung-Levy, Yuhui Zhang

llms agents fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 1

EgoCogNav: Cognition-aware Human Egocentric Navigation

arXiv:2511. 17581v3 Announce Type: replace Abstract: Modeling the cognitive and experiential factors of human navigation is central to deepening our understanding of human-environment interaction and to enabling safe social navigation and effective assistive wayfinding.

By Zhiwen Qiu, Ziang Liu, Wenqian Niu, Tapomayukh Bhattacharjee, Saleh Kalantari

multimodal

arXiv AI

Jul 1

Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models

arXiv:2606. 31846v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models offer a promising framework for robotic manipulation by connecting language instructions, visual observations, and continuous control.

By Lang Cao, Renhong Chen, Luyi Li, Peng Wang, Mofan Peng, Yitong Li

reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 1

Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

arXiv:2606. 31719v1 Announce Type: cross Abstract: In collaborative dialogue, shared perception does not guarantee shared interpretation.

By Nan Li, Albert Gatt, Massimo Poesio

llms multimodal safety

arXiv AI

Jul 1

RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization

arXiv:2606. 31694v1 Announce Type: cross Abstract: For robots manipulating open-world objects, tactile representations must generalize to unseen materials.

By Jingbo He, Michael F\"arber, Roberto Calandra

rag robotics multimodal

arXiv AI

Jul 1

REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

arXiv:2511. 17442v3 Announce Type: replace-cross Abstract: Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines.

By Binger Chen, Tacettin Emre B\"ok, Behnood Rasti, Volker Markl, Beg\"um Demir

llms agents computer-vision nlp multimodal benchmarks

arXiv AI

Jul 1

Gated Multi-Graph Fusion via Graph Attention Networks for Alzheimer's Disease Detection

arXiv:2606. 31186v1 Announce Type: cross Abstract: Spontaneous speech is a vital non-invasive biomarker for Alzheimer's Disease (AD), yet many systems overlook non-linear structural disruptions and clinical heterogeneity in pathological language.

By Jinyu Li, Xiao Wei, Bin Wen, Kai Li, Yuqin Lin, Xiaobao Wang, Longbiao Wang, Jianwu Dang

multimodal benchmarks

arXiv Machine Learning

Jul 1

ViTL: Temporal Logic-Guided Zero-Shot Natural Language Navigation via Vision-Language Models

arXiv:2606. 30696v1 Announce Type: cross Abstract: Enabling robots to follow natural language commands to complete zero-shot long-horizon tasks remains challenging.

By Kaier Liang, Hengde Dai, Cristian-Ioan Vasile

llms robotics multimodal

arXiv AI

Jul 1

Human-Agent Collaborative Paper-to-Page Crafting

arXiv:2510. 19600v2 Announce Type: replace-cross Abstract: In the quest for scientific progress, communicating research is as vital as the discovery itself.

By Qianli Ma, Siyu Wang, Yilin Chen, Yinhao Tang, Yixiang Yang, Chang Guo, Bingjie Gao, Zhening Xing, Yanan Sun, Zhipeng Zhang

agents multimodal benchmarks safety

arXiv AI

Jul 1

Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection

arXiv:2606. 30675v1 Announce Type: cross Abstract: Early detection of dementia through speech analysis offers a non-invasive screening alternative, but capturing both acoustic and linguistic biomarkers remains challenging.

By Olivier Jiyoun Jung, Jonghyeon Park, Myungwoo Oh

llms rag multimodal benchmarks

arXiv AI

Jul 1

Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images

arXiv:2606. 31394v1 Announce Type: cross Abstract: Artificial intelligence is transforming our capability to solve biological challenges.

By Jisung Park, Seohyeon Kang, Daeun Yoo, Eunsu Lee, Seoin Cho, Wooyeop Choi, Ian Choi, James R. Evan, Daesoo Kim, Sonia Gandhi, Minee L. Choi

multimodal safety

arXiv AI

Jul 1

WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models

arXiv:2606. 31672v1 Announce Type: cross Abstract: Despite rapid progress in interactive world models (IWMs), existing benchmarks evaluate action following only at trajectory level and ignore memory and interaction physics.

By Ting-Bing Xu, Jiacheng Sui, Zhe Gao, Kewei Shi, Wenjin Yang, Zhicheng Liu, Zhaoxu Sun, Mingchao Sun, Hongyu Pan, Fan Jiang, Mu Xu, Qi Fan, Yong Li, Baoquan Chen

multimodal benchmarks

arXiv AI

Jul 1

Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

arXiv:2606. 31711v1 Announce Type: new Abstract: Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) models.

By Yuanhao Ban, Tong Xie, Sohyun An, Yunqi Hong, Evan Frick, I-Hung Hsu, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

diffusion multimodal benchmarks

arXiv AI

Jul 1

A Modular Vision-Language-Action Robotics Framework for Indoor Environments

arXiv:2606. 31144v1 Announce Type: cross Abstract: This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions.

By Anindya Jana, Snehasis Banerjee, Arup Sadhu, Ranjan Dasgupta

llms rag agents robotics multimodal

arXiv AI

Jul 1

MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents

arXiv:2606. 31167v1 Announce Type: cross Abstract: VLA models have emerged as a powerful paradigm for transferring semantic knowledge from web-scale data to physical robotic control.

By Hao Sun, Yu Song, Shiyu Teng, Ziwei Niu, Yen-Wei Chen

rag agents robotics efficiency multimodal benchmarks

arXiv AI

Jul 1

TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

arXiv:2603. 29759v2 Announce Type: replace-cross Abstract: Recent advances in vision-language models (VLMs) have accelerated their application to indoor safety hazards assessment.

By Qiucheng Yu, Ruijie Xu, Mingang Chen Jianfeng Dong, Xin Tan

llms multimodal benchmarks safety

arXiv AI

Jul 1

MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

arXiv:2606. 31966v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored.

By Qingyun Liu, Jiwen Zhang, Jingyi Hu, Siyuan Wang, Zhongyu Wei

llms agents robotics multimodal benchmarks

arXiv AI

Jul 1

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

arXiv:2606. 31329v1 Announce Type: cross Abstract: Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation.

By Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

llms robotics multimodal benchmarks

Get the daily briefing