Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,393 stories · RSS feed

arXiv Machine Learning

Jun 16

Localized Kernel Projection Outlyingness: A Two-Stage Approach for Multi-Modal Outlier Detection

arXiv:2510. 24043v4 Announce Type: replace Abstract: This paper presents Two-Stage LKPLO, a novel multi-stage outlier detection framework that overcomes the coexisting limitations of conventional projection-based methods: their reliance on a fixed statistical metric and their assumption of a single data structure.

By Akira Tamamori

multimodal benchmarks

arXiv Machine Learning

Jun 16

Reinforcement Learning-Guided Retrieval with Soft Fusion for Robust Multimodal Imitation Learning under Missing Modalities

arXiv:2606. 15514v1 Announce Type: cross Abstract: Robotic systems perceive the world through multiple input modalities -- including visual camera streams and natural language instructions -- and must select appropriate actions based on these signals.

By Hassan Ismkhan, Hamid Bouchahcia

rag reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 16

Multi-Modal Spatio-Temporal Graph Neural Network with Mixture of Experts for Soil Organic Carbon Prediction

arXiv:2606. 16580v1 Announce Type: new Abstract: Top-soil organic carbon (SOC) prediction is fundamental to agricultural sustainability, land use policy and fertilization planning.

By Daniele Mos, Felipe Drummond, Anton Bossenbroek, Soufiane el Khinifri

rag fine-tuning multimodal

arXiv AI

Jun 16

Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

arXiv:2603. 24058v2 Announce Type: replace-cross Abstract: Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis.

By Han Sun, Qin Li, Peixin Wang, Min Zhang

llms agents robotics multimodal benchmarks safety

arXiv AI

Jun 16

CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions

arXiv:2512. 01095v2 Announce Type: replace-cross Abstract: We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their ability for textual reasoning over cyclical state transitions.

By Simon Kohaut, Daniel Ochs, Shun Zhang, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami

llms multimodal benchmarks

arXiv AI

Jun 16

Explainable deep learning improves human mental models of self-driving cars

arXiv:2411. 18714v3 Announce Type: replace-cross Abstract: Self-driving cars increasingly rely on deep neural networks to achieve human-like driving.

By Eoin M. Kenny, Akshay Dharmavaram, Sang Uk Lee, Tung Phan-Minh, Shreyas Rajesh, Yunqing Hu, Laura Major, Momchil S. Tomov, Julie A. Shah

agents robotics multimodal safety

arXiv AI

Jun 16

Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equation

arXiv:2512. 07212v3 Announce Type: replace Abstract: Imitation learning with diffusion models has advanced robotic control by capturing the multi-modal action distributions.

By Zhaoyang Liu, Mokai Pan, Zhongyi Wang, Kaizhen Zhu, Haotao Lu, Haipeng Zhang, Jingya Wang, Ye Shi

diffusion robotics multimodal benchmarks

arXiv AI

Jun 16

Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment

arXiv:2606. 16799v1 Announce Type: cross Abstract: Existing vision-language model (VLM)-based AI-generated image quality assessment (AIGIQA) methods suffer from a fundamental semantic-distortion dimensional conflict: monolithic representations optimized for semantic discrimination inherently entangle compositional understanding with low-level perceptual sensitivity, rendering them blind to fine-grained quality degradations.

By Zijie Meng

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 16

FlowMPC: Improving Flow Matching policies with World Models

arXiv:2606. 16286v1 Announce Type: cross Abstract: Flow Matching (FM) is a powerful approach for behavior cloning in multimodal action spaces [Jiang et al.

By Chandon Hamel

diffusion robotics multimodal

arXiv AI

Jun 16

MAGE-RAG: Multigranular Adaptive Graph Evidence for Agentic Multimodal RAG in Long-Document QA

arXiv:2606. 15906v1 Announce Type: cross Abstract: Long-document multimodal question answering requires a system to locate sparse evidence in long PDFs and integrate clues from text, tables, images, charts, and complex layouts.

By Yilong Zuo, Xunkai Li, Jing Yuan, Qiangqiang Dai, Hongchao Qin, Ronghua Li

rag agents nlp multimodal

arXiv AI

Jun 16

Post-Launch Capability Expansion of Vision-Language Models via Prompting for On-Orbit Spacecraft Inspection

arXiv:2606. 15427v1 Announce Type: cross Abstract: Spaceborne inspection systems often deploy perception models prior to launch, after which updating model weights or expanding fixed label sets becomes operationally impractical.

By Nicholas A. Welsh, Lennon J. Shikhman, Monty Nehru Attazs, Seemanthini K. Putane, Van Minh Nguyen, Ryan T. White

llms computer-vision multimodal

arXiv AI

Jun 16

Driving, Fast or Slow? Neuro-Symbolic Guidance for Motion Prediction in Multi-Modal Ground Mobility

arXiv:2606. 15251v1 Announce Type: cross Abstract: Accurate and interpretable motion prediction for heterogeneous traffic spaces, including pedestrians, bicycles, cars, and trucks, is essential for safe autonomous navigation.

By Simon Kohaut, Felix Divo, Julius Hahnewald, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami

agents multimodal benchmarks

arXiv AI

Jun 16

Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection

arXiv:2606. 15117v1 Announce Type: cross Abstract: The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both.

By Elham Abolhasani, Maryam Ramezani, Hamid R. Rabiee

robotics fine-tuning multimodal safety

arXiv AI

Jun 16

MatchLM2Lite: A Scalable MLLM-to-Lite Framework for Reproduced Content Identification

arXiv:2606. 14786v1 Announce Type: cross Abstract: Content moderation is critical for online video platforms to ensure content safety, protect creators, and sustain positive user experiences.

By Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Zirui Zhu, Kanchan Sarkar, Kun Xu

llms efficiency multimodal safety

arXiv AI

Jun 16

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

arXiv:2606. 14777v1 Announce Type: cross Abstract: Many moments in the real world do not wait for a user to ask.

By Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Haowen Hou, Zheming Liang, Congcong Wang, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang

llms agents multimodal

arXiv AI

Jun 16

OSGuard: A Benchmark for Safety in Computer-Use Agents

arXiv:2606. 15034v1 Announce Type: new Abstract: Computer-use agents are increasingly evaluated by whether they complete realistic desktop and web tasks.

By Mina Mohammadmirzaei, Jeffrey Flanigan

agents multimodal benchmarks safety

arXiv AI

Jun 16

MAND: Modality-Aware Novelty Detection for Open-World Egocentric Activity Recognition

arXiv:2603. 16970v2 Announce Type: replace-cross Abstract: Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding.

By Hyejeong Im, Wonseon Lim, Dae-Won Kim

efficiency multimodal benchmarks

arXiv Machine Learning

Jun 16

Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models

arXiv:2606. 15099v1 Announce Type: cross Abstract: Existing Vision-Language-Action (VLA) models predominantly rely on explicit Chain-of-Thought (CoT) reasoning to bridge perception and action.

By Dianqiao Lei, Lianlei Shan

reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jun 16

Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

arXiv:2505. 18227v4 Announce Type: replace-cross Abstract: In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks.

By Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, Marinka Zitnik

llms rag agents reinforcement-learning efficiency multimodal safety

arXiv AI

Jun 16

RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking

arXiv:2604. 20623v2 Announce Type: replace-cross Abstract: Traditional change detection identifies where changes occur, but does not explain what changed in natural language.

By Roie Kazoom, Yotam Gigi, George Leifman, Tomer Shekel, Genady Beryozkin

rag computer-vision multimodal benchmarks

Get the daily briefing