Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,972 stories · RSS feed

arXiv Machine Learning

Jun 30

TRACE: A Concept Bottleneck Model for Longitudinal 3D Glioblastoma Response Assessment

arXiv:2606. 30313v1 Announce Type: cross Abstract: Longitudinal glioblastoma response assessment requires comparing subtle tumor changes across MRI time points using structured clinical criteria such as RANO.

By Alia Tarek, Hamsa Saberr, Hamza Elghonemy, Youssef Afify, Tamer Basha, Omair Shahzad Bhatti, Abdulrahman M. Selim, Hasan Md Tusfiqur Alam Daniel Sonntag

multimodal benchmarks

arXiv AI

Jun 30

A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis

arXiv:2606. 29193v1 Announce Type: cross Abstract: LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data.

By Yuanhong Cai, Xiaohui Nie, Kanglin Yin, Changhua Pei, Yongqian Sun, Shenglin Zhang, Haibin Liu, Guiyang Liu, Xidao Wen, Fang Situ, Dan Pei

llms agents multimodal benchmarks

arXiv AI

Jun 30

When Stopping Fails: Rethinking Minimal Risk Conditions through Human-Interactive Autonomous Driving for Safe Transportation Systems

arXiv:2606. 29115v1 Announce Type: cross Abstract: Autonomous vehicles (AVs) are increasingly deployed in urban environments, yet their safety frameworks remain primarily designed around collision avoidance and minimal risk condition (MRC) behaviors such as slowing or stopping when uncertainty arises.

By Yash Tandon, Giovanni Tapia Lopez, Marcus Blennemann, Mohan Trivedi, Ross Greer

agents robotics multimodal safety

arXiv Machine Learning

Jun 30

Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs

arXiv:2606. 28401v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have shown strong performance in visual understanding, yet they still suffer from hallucinations, generating content that is not grounded in the image.

By Yunhun Nam, Jongheon Jeong

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 30

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

arXiv:2606. 29879v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning.

By Chen Yang, Yuhao Wei, Ze Xu, Ziheng Zou, Shuang Liang, Delin Ouyang, Lingfeng Qi, Jie Li, Guofa Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech

arXiv:2606. 30543v1 Announce Type: cross Abstract: With the proliferation of speech AI agents, understanding emotional entrainment in conversational interaction has become increasingly important.

By Sathvik Manikantan Napa Ugandhar, Hao Zhang, Alison Gunzler, Yuzhe Wang, Thomas Thebaud, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Vel\'azquez

rag agents fine-tuning multimodal

arXiv AI

Jun 30

Digitizing Coaching Intelligence: An Agentic Framework for Holistic Athlete Profiling using VLM and RAG

arXiv:2606. 28570v1 Announce Type: cross Abstract: Athlete assessment is a critical process for tracking physical progress and identifying elite talent.

By Deep Ghosal, Ishani Sen, Wazib Ansar, Amlan Chakrabarti

llms rag agents computer-vision multimodal safety data-engineering

arXiv AI

Jun 30

SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics

arXiv:2606. 29247v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models represent a promising direction for embodied intelligence in surgical robotics.

By Jiashuo Sun, Yue He, Wenxuan Liu, Tao Mao, Jiazheng Wang, Xiang Chen, Min Liu

diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders

arXiv:2606. 29888v1 Announce Type: new Abstract: Vision-language models map images and text into a joint embedding space.

By Chungpa Lee, Jihoon Kwon, Kyle Min, Jy-yong Sohn

llms rag multimodal safety

arXiv Machine Learning

Jun 30

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

arXiv:2606. 28406v1 Announce Type: new Abstract: Text-to-image and multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts.

By Davie Chen

diffusion computer-vision multimodal benchmarks

arXiv AI

Jun 30

SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

arXiv:2606. 21401v2 Announce Type: replace-cross Abstract: Agentic AI applications compose multiple model calls and tool executions, creating new scheduling challenges for GPU-CPU clusters.

By Yeqi Huang, Yanwei Ye, Guomin Chen, Wenhao Su, Bin Gong, Jialian Li, Zhan Lu, Yangshen Deng, Xuan Sun, Le Xu, Luo Mai

agents efficiency multimodal benchmarks

arXiv AI

Jun 30

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

arXiv:2603. 23559v2 Announce Type: replace-cross Abstract: GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices.

By Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang

llms agents multimodal benchmarks

arXiv AI

Jun 30

Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detector

arXiv:2606. 30196v1 Announce Type: cross Abstract: This paper offers an in-depth analysis of non-sequential multimodal sentence-level embeddings, with a particular focus on the SONAR model.

By Elys Allesiardo, Antoine Caubri\`ere, Valentin Vielzeuf

rag multimodal

arXiv AI

Jun 30

PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

arXiv:2606. 28344v1 Announce Type: cross Abstract: Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting.

By Yichuan Wang, Zhifei Li, Zirui Wang, Paul Teiletche, Lesheng Jin, Matei Zaharia, Joseph E. Gonzalez, Sewon Min

llms rag agents multimodal benchmarks

arXiv AI

Jun 30

CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

arXiv:2606. 29136v1 Announce Type: cross Abstract: Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames.

By Yu Li, Yuenan Hou, Yingmei Wei, Jiangming Chen, Yanming Guo

llms computer-vision multimodal benchmarks safety

arXiv AI

Jun 30

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

arXiv:2603. 29139v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks.

By Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Nathaniel Gorski, Jianxin Sun, Guoxi Liu, Helgi I. Ingolfsson, David Lenz, Hanqi Guo, Hongfeng Yu, Teja Leburu, Michael Molash, Bei Wang, Tom Peterka, Chaoli Wang, Shusen Liu

llms agents multimodal benchmarks

arXiv AI

Jun 30

Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL

arXiv:2604. 28123v3 Announce Type: replace-cross Abstract: The standard post-training recipe for large multimodal models (LMMs) applies supervised fine-tuning (SFT) on curated demonstrations followed by reinforcement learning with verifiable rewards (RLVR).

By Sudong Wang, Weiquan Huang, Xiaomin Yu, Zuhao Yang, Hehai Lin, Keming Wu, Chaojun Xiao, Chen Chen, Wenxuan Wang, Beier Zhu, Yunjian Zhang, Chengwei Qin

llms reinforcement-learning fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jun 30

Low-cost concept-based localized explanations: How far can we get with training-free approaches?

arXiv:2606. 29069v1 Announce Type: new Abstract: Concept-based Explainable AI (C-XAI) seeks human-understandable explanations grounded in semantic concepts, yet validation is limited by the scarcity of fine-grained concept annotations.

By Darian Fern\'andez-Guti\'errez, Rafael Bello, Marilyn Bello, Natalia D\'iaz-Rodr\'iguez

llms rag multimodal

arXiv AI

Jun 30

Hybrid Retriever Evolution for Multimodal Document Reasoning Agents

arXiv:2606. 29648v1 Announce Type: cross Abstract: Different retrievers, including lexical, semantic, and multimodal approaches, provide highly complementary strengths for multimodal document understanding, yet most systems combine them through fixed pipelines that cannot adapt to the demands of individual reasoning steps.

By Bohan Yao, Shruthan Radhakrishna, Vikas Yadav

agents nlp multimodal

arXiv AI

Jun 30

VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

arXiv:2606. 30645v1 Announce Type: cross Abstract: Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion.

By Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, Karen Liu

robotics multimodal

Get the daily briefing