Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,506 stories · RSS feed

arXiv AI
Sep 2

Risk-Aware Decision-Making for Autonomous Overtaking: A World Model-Based Mixture-of-Experts Framework

The paper introduces WM‑RMoE, a World Model‑based Risk‑aware Mixture‑of‑Experts framework for autonomous overtaking. It uses a learned latent dynamics model to perform multi‑step rollouts, evaluating cumulative risk at the trajectory level, and employs a hierarchical gating mechanism to coordinate long‑, short‑horizon, and rule‑based safety experts. A Gaussian Mixture Model preserves multimodal maneuver branches, improving robustness and preventing behavioral averaging, leading to better safety compliance, decision stability, and generalization in experiments.

By Yongzhi Liu, Sunan Zhang, Jinchang Xu, Jiawei Wang, Yushu Qiu, Chen Lv, Weichao Zhuang
arXiv AI
Sep 2

ADGNet: Asymmetric Dual-text Guided Network for Infrared Small Target Detection

ADGNet introduces an Asymmetric Dual-text Guided Network for infrared small target detection, addressing challenges of pixel-level methods and multimodal approaches that lack regional guidance. It employs an Asymmetric Dual-text Prompt (ADP) with an abstract target prompt and a detailed background prompt, and an Asymmetric Dual-Branch Interaction (ADBI) module to guide visual features separately, followed by an Adaptive Feature Aggregation (AFA) module for dynamic fusion. The authors also create an Asymmetric Image-Text Infrared (AITIR) dataset with asymmetric text annotations for three public datasets, and show that ADGNet outperforms 21 state‑of‑the‑art methods.

By Tongtong Wang, Mingzhu Xu, Chenglong Yu, Jing Wang, Xiaohui Lin, Weili Guan
arXiv AI
Sep 2

REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

REFACTOR‑VLA is a wake/sleep framework that learns reusable motor‑program abstractions for vision‑language‑action models. In its sleep phase, it clusters motor‑program fragments using a Behavioral‑Equivalence Kernel derived from rollouts of a latent world model, while its wake phase emits typed lambda terms that a library‑conditioned action decoder consumes. Experiments on LIBERO show that adding an auxiliary supervised contrastive loss during world‑model warm‑up significantly improves clustering quality, outperforming the strongest published baseline across all four task suites.

By Riyaaz Shaik, Chandru Venkataraman
arXiv AI
Sep 2

Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models

The study evaluates large language models for assessing suicide risk in Arabic crisis helpline calls, comparing Arabic and English models. Using de‑identified transcripts from Lebanon’s National Lifeline, the researchers fine‑tuned instruction‑tuned LLMs and transformer encoders, achieving a macro‑F1 of 81.19 and ROC‑AUC of 90.61 for high‑risk calls in Arabic, and 85.00/92.59 in English. The results show that high‑risk calls are more distinguishable than at‑risk calls, and translating to English does not degrade performance, indicating potential for operator‑facing tools.

By Linhai Ma, Rita El Hachem, Mahatab El Hajj, Lilian Ghandour, Samah Fodeh
arXiv AI
Sep 2

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

The paper investigates whether audio‑language models capture paralinguistic cues beyond spoken content. Using the Expresso dataset and four open‑source models, the authors trace how speaking style information is encoded in the late layers of the audio encoder but is degraded before reaching the final output. They find that some models are content‑driven while others are acoustic‑driven, revealing a gap between what is encoded and what is utilized in current audio‑language models.

By Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj
arXiv AI
Sep 2

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

The study investigates whether attention weights in Vision‑Language Models (VLMs) accurately reflect model reasoning for visual inputs. Using causal perturbation analysis, it identifies three distinct processing modes—Faithful‑Sufficient, Faithful‑Distributed, and Non‑Focal—indicating heterogeneous visual attention faithfulness. The research also shows that human‑annotated ground‑truth regions align with model attention in only about 60% of cases, highlighting a systematic divergence between model visual reliance and human intuition across VQA, document, and chart tasks.

By Xurui Song, Weishi Wang, Zhongqi Yue, Kuluhan Binici, Tao Bai, Hongxin Shao, Daniel Dahlmeier, Jun Luo
arXiv AI
Sep 2

X-SG$^2$S: Safe and Generalizable Gaussian Splatting with X-dimensional Watermarks

The paper introduces X‑SG$^2$S, a feed‑forward framework that embeds 1D to 3D watermarks into 3D Gaussian Splatting (3DGS) scenes without altering the original rendering pipeline. It splits watermark messages into patches, uses a self‑adaptive gate to choose injection points, and an XD injection head to embed multi‑modal messages into sorted 3DGS points. A learnable gate and XD‑extraction heads recover the hidden messages, achieving robust watermarking with minimal interference to scene fidelity.

By Zihang Cheng, Wentao Bao, Huiping Zhuang, Chun Li, Xin Meng, Ziqian Zeng, Cen Chen, Ming Li, F. Richard Yu
arXiv Computation and Language
Sep 2

IntroConformal: Conformal Factuality Guarantees for Large Vision-Language Models via Introspective Signals

IntroConformal introduces a training‑free Conformal Risk Control framework that offers finite‑sample, distribution‑free factuality guarantees for Large Vision‑Language Models. It uses introspective signals—layer‑wise semantic stability and verification probability derived from the model’s own hidden states—to assess claim factuality. Experiments across multiple LVLM architectures show that IntroConformal meets the conformal risk guarantee while reducing abstention and matching or surpassing external verifier baselines in claim‑level discrimination.

By Md. Atabuzzaman, Christian Alexander, Chris Thomas
arXiv AI
Sep 2

Conversation Coach: A Voice-enabled AI System that Helps Practice Difficult Workplace Conversations

Conversation Coach is a voice‑first AI system that lets managers rehearse difficult workplace conversations in a realistic spoken format. It tackles low‑latency interaction, adaptive bot personalities that simulate various employee types, and personalized feedback on content and policy compliance. The authors compare an end‑to‑end speech‑to‑speech model with a cascaded approach, finding the former offers lower latency and cost, while the cascaded model provides better reasoning for coaching quality, and they deployed the cascaded architecture to 40,000+ managers over six months.

By Fanyou Wu, Suraj Maharjan, Ainur Yessenalina, Dennis Xu Chen, Rahul Srivastava, Srinivasan H. Sengamedu
arXiv AI
Sep 2

Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy

The paper investigates a failure mode in multimodal large language models called multimodal contextual sycophancy, where external text can override conflicting image evidence. A diagnostic set of 998 cases independently varies visual evidence, commonsense priors, and external text to probe when this failure occurs. Experiments across six models show that a System‑2 Visual Arbitration (S2VA) approach, which withholds text from the visual witness, significantly improves performance over direct witness reports, with the best information boundary varying by model and context source.

By Yi-Cheng Lai, Hen-Hsen Huang
arXiv Computer Vision
Sep 2

Feed-Forward Multi-view Multi-person Reconstruction with Contrastive Human-Aware 3D Representation

The paper introduces a top‑down approach for multi‑person 3D reconstruction from multiple views, using a unified, instance‑centric human‑aware 3D space. Observations from different cameras are lifted into this shared space where geometry, appearance, and semantic cues are jointly encoded, and a spatial contrastive learning strategy aligns features of the same person across views while separating different individuals. The method then regresses SMPL parameters from 3D tokens in a feed‑forward manner, achieving robust, accurate, and efficient reconstruction even under severe occlusions.

By Yuanwang Yang, Buzhen Huang, Zongxuan Ren, Jing Huang, Kun Li
arXiv AI
Sep 2

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

arXiv:2609.00551v1 Announce Type: cross Abstract: Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, s...

By Yijun Chen, Yaqi Zheng, Yanya Li, Boyi Xiao, Buqiang Xu, Shuofei Qiao, Jizhan Fang, Xinle Deng, Yunzhi Yao, Xuehai Wang, Liuxin Zhang, Hui Li, Huajun Chen, Shumin Deng