Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,396 stories · RSS feed

arXiv AI

Jul 1

TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

arXiv:2603. 29759v2 Announce Type: replace-cross Abstract: Recent advances in vision-language models (VLMs) have accelerated their application to indoor safety hazards assessment.

By Qiucheng Yu, Ruijie Xu, Mingang Chen Jianfeng Dong, Xin Tan

llms multimodal benchmarks safety

arXiv AI

Jul 1

ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models

arXiv:2606. 30682v1 Announce Type: cross Abstract: Recent advances in language--audio retrieval have been largely driven by contrastive dual-encoder architectures that align audio and text in a shared embedding space.

By Fengjie Lu, Chenang Jiang, Jiarui Hai, Helin Wang, Aaron Yee

llms rag nlp multimodal benchmarks

arXiv Machine Learning

Jul 1

Localized Conformal Prediction for Image Classification with Vision-Language Models

arXiv:2606. 31577v1 Announce Type: cross Abstract: Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees.

By Cl\'ement Fuchs, Tim Bary, Beno\^it Macq

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jul 1

InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context

arXiv:2603. 05353v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) for long-context question answering is bottlenecked by inference-time prefilling over large retrieved contexts.

By Xin Teng, Canyu Zhang, Shaoyi Zheng, Danyang Zhuo, Tianyi Zhou, Shenji Wan

llms rag nlp multimodal benchmarks

arXiv Machine Learning

Jul 1

Medical Image Spatial Grounding with Semantic Sampling

arXiv:2603. 14579v3 Announce Type: replace-cross Abstract: Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos.

By Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura, Mingrui Yang, Xiaojuan Li, Vipin Chaudhary

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jul 1

EgoCogNav: Cognition-aware Human Egocentric Navigation

arXiv:2511. 17581v3 Announce Type: replace Abstract: Modeling the cognitive and experiential factors of human navigation is central to deepening our understanding of human-environment interaction and to enabling safe social navigation and effective assistive wayfinding.

By Zhiwen Qiu, Ziang Liu, Wenqian Niu, Tapomayukh Bhattacharjee, Saleh Kalantari

multimodal

arXiv Machine Learning

Jul 1

Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

arXiv:2606. 31717v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks.

By Shuai Yuan, Sudong Cai, Bingzhi Chen, Shuyuan Zheng, Chuan Xiao, Makoto Onizuka, Rui Mao

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 1

Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models

arXiv:2606. 31846v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models offer a promising framework for robotic manipulation by connecting language instructions, visual observations, and continuous control.

By Lang Cao, Renhong Chen, Luyi Li, Peng Wang, Mofan Peng, Yitong Li

reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 1

GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation

arXiv:2603. 26266v3 Announce Type: replace Abstract: Large vision-language models have endowed GUI agents with strong general capabilities for interface understanding and interaction.

By Rui Xie, Zhi Gao, Chenrui Shi, Zirui Shang, Lu Chen, Qing Li

llms rag agents multimodal safety

arXiv AI

Jul 1

Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue

arXiv:2606. 31719v1 Announce Type: cross Abstract: In collaborative dialogue, shared perception does not guarantee shared interpretation.

By Nan Li, Albert Gatt, Massimo Poesio

llms multimodal safety

arXiv AI

Jul 1

A Modular Vision-Language-Action Robotics Framework for Indoor Environments

arXiv:2606. 31144v1 Announce Type: cross Abstract: This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions.

By Anindya Jana, Snehasis Banerjee, Arup Sadhu, Ranjan Dasgupta

llms rag agents robotics multimodal

arXiv AI

Jul 1

ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

arXiv:2606. 31054v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image.

By Zhiyuan Yao, Zheren Fu, Zhixiao Zheng, Jiajun Li, Yi Tu, Zhendong Mao

llms diffusion reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 1

LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music

arXiv:2606. 31158v1 Announce Type: cross Abstract: The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics.

By Snehasis Banerjee, Ranjan Dasgupta

llms robotics multimodal

arXiv AI

Jul 1

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

arXiv:2606. 31329v1 Announce Type: cross Abstract: Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation.

By Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

llms robotics multimodal benchmarks

arXiv AI

Jul 1

Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

arXiv:2606. 31407v1 Announce Type: cross Abstract: Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions.

By Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W. Verjans, Vu Minh Hieu Phan

llms rag multimodal benchmarks

arXiv Machine Learning

Jul 1

Structured SIR: Efficient and Expressive Importance-Weighted Inference for High-Dimensional Image Registration

arXiv:2603. 17415v2 Announce Type: replace-cross Abstract: Image registration is an ill-posed dense vision task, where multiple solutions achieve similar loss values, motivating probabilistic inference.

By Ivor J. A. Simpson, Neill D. F. Campbell

multimodal

arXiv AI

Jul 1

Gated Multi-Graph Fusion via Graph Attention Networks for Alzheimer's Disease Detection

arXiv:2606. 31186v1 Announce Type: cross Abstract: Spontaneous speech is a vital non-invasive biomarker for Alzheimer's Disease (AD), yet many systems overlook non-linear structural disruptions and clinical heterogeneity in pathological language.

By Jinyu Li, Xiao Wei, Bin Wen, Kai Li, Yuqin Lin, Xiaobao Wang, Longbiao Wang, Jianwu Dang

multimodal benchmarks

arXiv AI

Jul 1

MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

arXiv:2606. 31966v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored.

By Qingyun Liu, Jiwen Zhang, Jingyi Hu, Siyuan Wang, Zhongyu Wei

llms agents robotics multimodal benchmarks

arXiv AI

Jul 1

Agentic AI Enhances Physician Trust in Clinical Decision Making

arXiv:2606. 30658v1 Announce Type: cross Abstract: Medical AI has shifted from reasoning to agentic AI, a new paradigm that autonomously invokes external tools during reasoning, rendering intermediate reasoning steps and tool outputs transparent to users.

By Zhiling Yan, Zhe Fang, David J King, Ann Pongsakul, Eashan Adhikarla, Hui Ren, Sunyang Fu, Quanzheng Li, Lifang He, Xiang Li, Hongfang Liu, Yonghui Wu, Lichao Sun

agents multimodal

arXiv Machine Learning

Jul 1

On Optimizing Multimodal Jailbreaks for Spoken Language Models

arXiv:2603. 19127v2 Announce Type: replace Abstract: As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone while introducing an expanded attack surface.

By Aravind Krishnan, Karolina Sta\'nczak, Dietrich Klakow

llms multimodal benchmarks safety

Get the daily briefing