← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,981 stories · RSS feed

arXiv Computation and Language
Sep 21

Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

The paper introduces Omni Demand Understanding (ODU), a benchmark designed to test whether multimodal models can infer a user's underlying demand from complex audio‑visual interactions. ODU requires models to detect the presence of a demand and infer intent using multimodal and conversational context, evaluated across single‑turn and multi‑turn scenarios. The authors built ODU‑Bench through a taxonomy‑guided approach, agentic video generation, and human‑recorded interactions, and found that even top models like Gemini 3.1 Pro recover only 44.7% of key information, with many models exhibiting high false‑trigger rates.

By Qi Chen, Yunfei Chu, Haolin He, Yifan Yang, Zihan Liu, Yuxuan Wang, Ziyang Ma, Ruiyang Xu, Meng Gao, Yinsong Yan, Ling Wang, Hui Wang, Wen Huang, Yiheng Chen, Guanrou Yang, Qiuqiang Kong, Jin Xu, Xie Chen
llmsagentsdiffusionmultimodalbenchmarks
More like this →
arXiv Computation and Language
Sep 21

Sometin Beta Pass Notin: Improving Multilingual ASR for Nigerian Languages via Knowledge Distillation

The paper presents Sometin Beta Pass Notin (SBPN), a multilingual ASR framework for Nigerian languages that uses a two‑stage knowledge‑distillation approach. First, student‑teacher distillation from monolingual models is conditioned on language‑specific N‑gram language models; second, iterative self‑improvement with pseudo‑labelled data further refines accuracy. The method reduces relative WER by 29% over monolingual baselines and outperforms state‑of‑the‑art multilingual models on Common Voice and FLEURS benchmarks for Yoruba, Hausa, Igbo, Nigerian Pidgin, and Nigerian English.

By Sewade Ogun
llmsefficiencymultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 21

From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists

The paper investigates how general vision‑language models (VLMs) develop specialized optical character recognition (OCR) capabilities. By applying a causal intervention protocol, the authors identify sparse, stable OCR‑head sets in several VLMs and show that these heads largely overlap with textual retrieval/copy heads found in general VLMs. The study concludes that full‑sequence OCR functions as a dense multimodal copy‑and‑paste mechanism, and that when a VLM is fine‑tuned for OCR, it largely preserves the same head identities while redistributing their functional and causal strengths.

By Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen, Jeffrey Tiong Jee Hui
llmscomputer-visionmultimodal
More like this →
arXiv Computer Vision
Sep 21

GALA: Geometry-Aware Latent Action Modeling for Vision-Language-Action Model Pretraining across Embodiments

The paper introduces GALA, a Geometry-Aware Latent Action modeling framework that enhances image-based latent actions with 3D end‑effector motion. It proposes the Unified End‑effector Motion Representation (UEMR) to preserve fine‑grained motion while improving cross‑embodiment generalizability. Experiments show GALA effectively models generalizable fine‑grained motions across embodiments, achieving high success rates in RoboCasa-GR1 and real‑world tasks.

By Yichen Liu, Puzhen Yuan, Xiang Zhu, Yanjiang Guo, Jianyu Chen
roboticsmultimodal
More like this →
arXiv Computer Vision
Sep 21

Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment

Traits Run Deeper introduces a personality assessment framework that tailors multimodal fusion to each trait dimension. It comprises a Multimodal Foundation Representation module that uses psychology-informed semantic templates, a Trait-Specific Modality Fusion module that asymmetrically fuses modalities to reduce cross‑modal interference, and a Distribution‑Calibrated Personality Regression module that corrects label imbalance. The approach achieves a ~25% reduction in mean squared error on the AVI Challenge 2026 validation set and wins the Personality Assessment Track.

By Jia Li, Qian Chen, Wei Wang, Xinyu Li, Zhenzhen Hu, Dongsheng Shao, Richang Hong, Meng Wang
multimodalsafety
More like this →
arXiv Computer Vision
Sep 21

SkillIR: Evolving Scene-Aware Skills for Agentic Image Restoration

SkillIR is a skill-guided framework for agentic image restoration that represents restoration experience as degradation-centered action evidence rather than full tool-use trajectories. It consolidates context-dependent action outcomes into scene-aware restoration skills, guiding one bounded action at a time within a verified residual-state loop. Experiments on synthetic and real-world multi-degradation datasets show that SkillIR improves restoration quality and enables more reliable and effective tool use.

By Jie Shao, Shengkai Hu, Xu Zhang, Beihang Song, Yongcheng Jing, Xu Wu, Jun Wan
agentsmultimodal
More like this →
arXiv Machine Learning
Sep 21

Reading Less While Writing: A Closed-Form Bandwidth Dial for Streaming Multimodal Decoders

The paper introduces ZENDAYA, a closed‑form bandwidth dial that replaces fixed wait‑k rules in streaming multimodal decoders with a single continuous parameter γ. This schedule ties the visible source prefix to generation progress and the input’s predicted length, allowing a single offline decoder to serve as both offline and real‑time streaming models. Experiments on video‑to‑text and audio‑to‑text tasks show that reading less of the source can improve text quality, with significant METEOR gains across three public corpora.

By Yasir Mehmood, Kashif Javed
multimodal
More like this →
arXiv Computer Vision
Sep 21

ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation

ULTRA is a unified framework for autonomous humanoid whole-body locomotion and manipulation that overcomes limitations of prior methods by combining a physics-driven neural retargeting algorithm with a multimodal controller. The retargeting algorithm translates large-scale motion capture data into physically plausible humanoid motions, while the controller learns to handle both dense motion references and sparse task specifications using a range of sensory inputs, from accurate motion-capture states to noisy egocentric vision. In simulation and on a real Unitree G1 humanoid, ULTRA demonstrates improved generalization and robustness, enabling coordinated whole-body behavior from sparse intent without relying on test-time reference motions.

By Xialin He, Sirui Xu, Xinyao Li, Runpei Dong, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui
agentsdiffusionreinforcement-learningroboticsfine-tuningmultimodal
More like this →
arXiv AI
Sep 21

DiaVLo: Diagnosing Behaviours of Vision-Language Models

DiaVLo is a diagnostic framework for vision‑language models (VLMs) that uses human curation and VLM generation to create specifications of desired and observed behaviours, revealing potential misalignments. It also offers causal estimates to pinpoint the most influential concepts driving VLM behaviour. Experiments on several open‑source VLMs under classification and generation tasks show that DiaVLo’s behaviour labels correlate with model performance and illuminate how VLMs perceive, organise, and prioritise concepts.

By Lorenzo Corti, Jie Yang
llmsmultimodal
More like this →
arXiv AI
Sep 21

Offline Multimodal Large Language Models for Decision Support in Air Operations

The paper explores offline large language models as decision‑support tools for air operations, where analysts must work with doctrine and imagery under limited connectivity. It presents a modular retrieval‑augmented architecture that handles both text and image inputs from technical manuals and can operate without Internet access. A pilot study with four Brazilian Air Force image analysts shows the system matches human performance on doctrinal knowledge tests while completing the task in a fraction of the time, and it highlights the high cognitive load of manual target reporting.

By Joao P. A. Dantas, Jelton A. Cunha, Gabriel Dietzsch
llmsragmultimodal
More like this →