← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,098 stories · RSS feed

arXiv Computation and Language
Sep 15

Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluatio

The paper investigates how to evaluate generative audio large language models (Audio‑LLMs) on known closed‑set tasks by separating the decision to call a generative model from the use of acoustic evidence. It introduces a controlled call‑decision framework where a policy can choose between a transcript label, encoder evidence from CLAP, AST, or WavLM, or a generative call to Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio, and measures the impact of generative calls on accuracy. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑based controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).

By Mengzhe Geng
llmsmultimodalbenchmarks
More like this →
arXiv Computation and Language
Sep 15

Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth

The paper investigates how vision‑language models (VLMs) read exact values from vertical bar charts. Using controlled counterfactual activation patching on Qwen2.5VL‑7B‑Instruct and InternVL3.5‑8B, the authors find that the bar‑top region contributes more to answer accuracy than the bar body, and that legend and series states lose recoverability earlier than geometry and axis‑scale states. They also show that prompt‑series positions mediate legend influence and that the two models differ in how they combine geometry and scale information from separate image donors.

By Tianhao Niu, Qingfu Zhu, Wanxiang Che
llmsmultimodal
More like this →
arXiv Computation and Language
Sep 15

Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models

The paper introduces a lightweight ASR head that can be added to full‑duplex speech‑to‑speech models, enabling real‑time user transcription without major architectural changes. The method adds only a few parameters and preserves full‑duplex conversational features such as turn‑taking and barge‑in. Experiments show a streaming WER of 10.21% within the duplex framework and 7.73% when trained as a standalone ASR model, matching state‑of‑the‑art performance.

By Ke Hu, Nourchene Ferchichi, Edresson Casanova, Ankita Pasad, Elena Rastorgueva, Chen Chen, Nithin Rao Koluguri, Piotr Zelasko, Yifan Peng, Hainan Xu, Zhehuai Chen, Boris Ginsburg
agentsmultimodalbenchmarks
More like this →