← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv Computation and Language
Sep 10

LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios

LogiScope‑VQA is a new benchmark dataset for evaluating vision‑language models in logistics hazard identification. It contains 2,476 images, 2,918 videos, and 10,274 VQA pairs drawn from real industrial warehouses, covering 18 core objects and 20 risk types across 39 subtasks. Experiments show that even advanced proprietary models lag behind human experts, highlighting a significant gap in perception, understanding, and reasoning for industrial safety.

By Hanjing Zhou, Mingze Yin, Ying Lian, Jun Ma, Chang-Yu Hsieh, Yanbing Zhou
llmsmultimodalbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 10

NOPE-HYPE: A Structured Simulation Workflow for Robust Speech-to-Text Across Diverse Acoustic Environments

NOPE-HYPE is a structured training workflow that integrates a controllable environment simulator, a coverage‑optimal reduction of Power Spectral Density templates, and a concise hyperparameter search over simulator settings. The authors demonstrate that noise generated by the simulator can match the performance of balanced real‑noise training for Whisper and SeamlessM4T models. They also provide principled environment prototype sets and practical default simulator configurations derived from a 27‑run hyperparameter sweep.

By Niramay M. Patel, Bibek Behera, Raksha Sharma
multimodal
More like this →
arXiv Computation and Language
Sep 10

Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization

arXiv:2609.10410v1 Announce Type: new Abstract: The growing complexity of content moderation policies presents a critical challenge for their consistent operationalization. While foundation models po...

By Ayan Majumdar, Shounak Paul, Pushpdeep Singh, Ines Abdelaziz, Sayeh Jarollahi, Seungeon Lee, Krishna P. Gummadi, Ingmar Weber, Abhisek Dash
llmsmultimodalbenchmarks
More like this →
arXiv AI
Sep 10

RevalExo: A Functional Daily-Activity Benchmark for Inertial and Visual Locomotion Mode Recognition in Older Adults and Clinical Cohorts

RevalExo is a new benchmark for locomotion mode recognition that focuses on functional daily activities performed by older adults and clinical cohorts. It includes 27 participants from three groups—healthy older adults, stroke survivors, and older adults with probable sarcopenia—recorded with lower-body IMUs and, for a subset, synchronized egocentric video. The dataset offers 10.1 hours of frame‑level annotations across 11 locomotion modes, and the authors evaluate unimodal, multimodal, cross‑population, and cross‑modal recognition challenges, finding that sensor fusion improves performance but transitions and generalization remain difficult.

By Diwas Lamsal, Juha Carlon, Reinhard Claeys, Maxim Yudayev, Louis Flynn, Tom Verstraten, David Beckw\'ee, Eva Swinnen, Mihai B\^ace, Bart Vanrumste, Benjamin Filtjens
roboticsmultimodalbenchmarks
More like this →
arXiv Machine Learning
Sep 10

I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models

The paper introduces an interventional protocol to assess how vision‑language models (VLMs) explain the impact of missing modalities on their predictions. By comparing the models’ self‑explanations with actual changes observed after restoring missing inputs, the study finds that VLMs routinely overstate the sufficiency of available evidence and underestimate the effect of adding back missing modalities. Across eight open‑weight VLMs and four tasks, the discrepancy between predicted and realized changes is substantial, revealing systematic mischaracterization of modality dependence.

By Aydin Javadov, Daniel Schoess, Florian von Wangenheim
llmsmultimodal
More like this →
arXiv AI
Sep 10

FastE: Readout-Triggered Token Compression for LLM Embedding Inference

FastE is a training‑free, plug‑and‑play method that compresses token prefixes in large language model (LLM) embedding inference. It uses a shared fixed threshold on batch‑mean readout‑prefix alignment to decide when to compress and ranks prefix states by readout attention scores to keep the most important ones. Experiments on Qwen3‑Embedding models show that FastE can cut decoder‑backbone FLOPs by over 40% while preserving more than 99% of the original ranking quality across multiple benchmarks and tasks.

By Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
llmsragmultimodalbenchmarkssafety
More like this →
arXiv AI
Sep 10

Knowledge-Guided Vision-Language Inference for Image-Based Urban Flood Depth Estimation

The paper introduces FloodVision, a knowledge-guided vision‑language framework that estimates urban flood depth from a single RGB image. It combines a general‑purpose vision‑language model with FloodKG, a domain knowledge base that encodes canonical object dimensions and component landmarks to promote component‑level reasoning. On 654 crowdsourced New York flood images, FloodVision cuts mean absolute error from 15.62 cm to 8.75 cm and median error from 14.35 cm to 7.75 cm, outperforming the VLM‑only baseline in 69.3 % of cases.

By Zhangding Liu, Neda Mohammadi, John E. Taylor
llmsragcomputer-visionmultimodal
More like this →
arXiv AI
Sep 10

TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

TimeBlind is a diagnostic benchmark designed to evaluate fine‑grained spatio‑temporal compositionality in video large language models (LLMs). It categorizes temporal understanding into three levels—atomic event recognition, event property characterization, and reasoning about event interdependencies—and uses a minimal‑pairs paradigm where video pairs share identical static content but differ only in temporal structure. Across 20 state‑of‑the‑art MLLMs tested on 600 curated instances, the best model achieved only 48.2% instance accuracy, far below human performance of 98.2%, highlighting a reliance on static visual shortcuts rather than true temporal reasoning.

By Baiqi Li, Kangyi Zhao, Ce Zhang, Chancharik Mitra, Jean de Dieu Nyandwi, Gedas Bertasius
llmsroboticsmultimodalbenchmarks
More like this →
arXiv AI
Sep 10

Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation

SCOUT is a concept‑grounded multimodal transformer that generates whole‑slide pathology reports by integrating local histological patterns, whole‑slide context, and expert‑curated diagnostic concepts. It uses evolving visual representations and recursively updated slide‑ and concept‑conditioned representations, with separate attention pathways during decoding that are fused adaptively for each token. Evaluated on TCGA‑BRCA, HistAI, and REG‑2025, SCOUT outperformed existing methods, improving BLEU, METEOR, and ROUGE‑L scores and raising the Clinical Report Quality Score on REG‑2025.

By Suryakant Singh, Saarthak Kapse, Joel Saltz, Prateek Prasanna
llmsmultimodal
More like this →