Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,414 stories · RSS feed

arXiv AI

Jun 2

Spatial Representation Learning Beyond Pixels: Unifying Raster Data and Vector Semantics for Human-Centric Geospatial Foundation Models

arXiv:2606. 02374v1 Announce Type: new Abstract: Earth Observation (EO) has fundamentally transformed the monitoring of environmental processes and human activities up to planetary scale.

By Steffen Knoblauch, Hao Li, Gengchen Mai, Konstantin Klemmer, Song Gao, WenWen Li

rag multimodal

arXiv AI

Jun 2

PropLLM: Propagation-Aware Scene Reconstruction for Network Fault Diagnosis

arXiv:2606. 00582v1 Announce Type: new Abstract: Network faults propagate layer by layer along topology and protocol dependencies, yet operations systems typically observe only symptomatic alerts at the tail end of propagation chains, where distinct root-cause faults may produce highly similar end-point symptoms.

By Zongzong Wu, Ming Zhao, Fengxiao Tang, Nei Kato

llms multimodal safety

arXiv Machine Learning

Jun 2

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.

By Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 2

Jailbreaking Multimodal Large Language Models using Multi-Clip Video

arXiv:2606. 02111v1 Announce Type: cross Abstract: As multimodal large language models (MLLMs) have advanced to process video inputs, concerns have emerged about their potential for malicious misuse.

By Choongwon Kang, Seungjong Sun, Hyunmin Jun, Jang Hyun Kim

llms multimodal safety

arXiv AI

Jun 2

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.

By Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

llms multimodal benchmarks safety

arXiv AI

Jun 2

GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval

arXiv:2606. 00775v1 Announce Type: cross Abstract: Video Moment Retrieval (VMR) task requires accurately localizing temporal boundaries aligned with natural language queries, but many models suffer from a misalignment between continuous surrogate losses and non-differentiable metrics, leading to optimization stagnation during the late stages of training and trapping boundary predictions in suboptimal solutions.

By Shihang Zhang, Mingjin Kuai, Ye Wei, Zhen Zhang, Wei Ji

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

V-LynX: Token Interface Alignment for Video+X LLMs

arXiv:2606. 00508v1 Announce Type: cross Abstract: This study introduces an intriguing phenomenon in Video LLMs: rather than merely translating frames into textual embeddings, Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture.

By Jungin Park, Jiyoung Lee, Kwanghoon Sohn

llms rag multimodal benchmarks safety

arXiv AI

Jun 2

Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty

arXiv:2606. 00670v1 Announce Type: cross Abstract: Face-to-face speech comprehension is inherently multimodal, integrating acoustic signals with visible articulation, facial expression, head motion, and other socially relevant cues.

By Zhou Yang, Yueyi Yang

multimodal

arXiv AI

Jun 2

VESTA: Visual Exploration with Statistical Tool Agents

arXiv:2606. 00384v1 Announce Type: new Abstract: Fitting quantitative models to data is a central step in scientific workflows, yet it remains one of the least automated.

By William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li

llms agents multimodal benchmarks

arXiv AI

Jun 2

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

arXiv:2603. 00171v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details.

By Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li, Man Zhou, Chengjun Xie, Haoxuan Che, Xuanhua He, Jie Zhang

llms multimodal benchmarks

arXiv AI

Jun 2

DraDDP: A Multimodal Multi-Party Dialogue Discourse Parsing Dataset

arXiv:2606. 00012v1 Announce Type: cross Abstract: Multi-party dialogue discourse parsing aims to identify dependency structures and relation types between utterances in conversations.

By Shannan Liu, Peifeng Li, Yaxin Fan, Qiaoming Zhu

multimodal benchmarks

arXiv AI

Jun 2

Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation

arXiv:2606. 01897v1 Announce Type: new Abstract: Traditional Video Quality Assessment (VQA) focuses narrowly on aesthetic fidelity, overlooking the complex social dynamics that define quality in User-Generated Content (UGC).

By Tianjiao Li, Kai Zhao, Xiang Li, Yang Liu, Huyang Sun

reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence

arXiv:2606. 02463v1 Announce Type: cross Abstract: In 3D environments, Embodied Agents answer spatially relevant questions through reasoning from a mixture of modalities including natural language, RGB images, point clouds, depth maps and camera poses.

By Hilton Raj, Vishnuram AV

llms rag agents robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs

arXiv:2606. 00813v1 Announce Type: cross Abstract: Safety alignment in LLMs does not improve monotonically across model generations.

By Subhadip Mitra

llms multimodal benchmarks safety

arXiv AI

Jun 2

APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

arXiv:2601. 21444v2 Announce Type: replace-cross Abstract: The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs).

By Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu

rag multimodal

arXiv Machine Learning

Jun 2

Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning

arXiv:2606. 00851v1 Announce Type: cross Abstract: Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues.

By Sukru Samet Dindar, Riki Shimizu, Xilin Jiang, Nima Mesgarani

multimodal safety

arXiv Machine Learning

Jun 2

TLG: Temporal-Logic Grounding for Video Question Answering via Source-Annotation Reconstruction and Category-Targeted Reasoning

arXiv:2606. 01591v1 Announce Type: cross Abstract: The TimeLogic Challenge evaluates formal temporal-logic reasoning over video - 16 operators (before, after, until, since, always, co-occur, ordering, ...

By Ali Alavi

llms nlp multimodal benchmarks

arXiv AI

Jun 2

StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement

arXiv:2606. 00267v1 Announce Type: cross Abstract: Video world models (WMs) have shown promise for policy evaluation and improvement by imagining realistic future observations conditioned on ego-robot actions.

By Junwon Seo, Sushant Veer, Ran Tian, Wenhao Ding, Apoorva Sharma, Karen Leung, Edward Schmerling, Marco Pavone, Andrea Bajcsy

llms agents diffusion robotics multimodal benchmarks

arXiv AI

Jun 2

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

arXiv:2606. 01992v1 Announce Type: cross Abstract: Industrial anomaly detection has historically been a unimodal task.

By Stefano Samele, Eugenio Lomurno, Teodora Jovanovic, Sanjay Shivakumar Manohar, Alberto Crivellaro, Matteo Matteucci

llms rag multimodal benchmarks

arXiv AI

Jun 2

Continuous Reasoning for Vision-Language-Action

arXiv:2606. 00229v1 Announce Type: cross Abstract: Natural language is a powerful reasoning medium for language and vision-language models, but it is mismatched to the granularity of continuous control.

By Yueh-Hua Wu, Tatsuya Matsushima, Kei Ota

llms robotics multimodal

Get the daily briefing