The paper investigates how large audio‑language models (LALMs) hallucinate objects by affirming sounds that are not present. Using a black‑box causal test, the authors isolate evidence regions via CLAP similarity, remove them with spectral gating, and compare against random windows across a 2×2×2 grid on three LALMs. The study finds that evidence localization improves detection over random removal, that raw detection scores are model‑dependent, and that a contrastive decoding strategy combining blank and language‑prior cues boosts Qwen2‑Audio accuracy from 57.1 % to 71.9 %, outperforming Audio‑Aware Decoding alone.
The study investigates where a large audio language model (LALM) derives its refusal responses to harmful spoken requests. By applying causal interventions at the audio-to-language-model interface and within the language-model residual layers, the authors find that the primary influence on the refusal margin comes from mid-to-late layers of the text language model rather than the audio front end. Ablations of the audio interface have minimal effect, while zeroing the encoder output still reduces the margin, indicating the audio pathway remains active but is not the main source of refusal decisions.
By Leonardo Haw-Yang Foo, Hung-yi Lee
The paper presents the first music‑specific, layer‑wise empirical study of hallucination in audio‑language models, framing it as a hierarchical perceptual grounding failure across five layers: sound events, temporal properties, tonal attributes, style, and emotion. It introduces MuseDiag, a diagnostic framework that evaluates nine models and finds universal vocal misperception, significant tonal perception differences, and identifies Audio‑Flamingo‑3 as the most stable model. The study also proposes two training‑free mitigation methods, ADD‑M and TPA, which reduce hallucination in probing but show variable effectiveness in free‑form generation, highlighting the need for multi‑paradigm evaluation.
By Yu Liu, Jiahui Liu, Zhilin Liu, Cong Cao, Fangfang Yuan, Yuling Yang, Pin Xu, Yanbing Liu
arXiv:2608. 19211v1 Announce Type: cross Abstract: Human speech is richly expressive, with prosody carrying linguistic and emotional information beyond the lexical content.
By Linkai Peng, Baorian Nuchged
The paper investigates how to evaluate audio‑language models by separating the use of acoustic evidence from the need to invoke a generative audio model. Using a controlled call‑decision framework, the authors compare policies that rely on transcript labels, encoder outputs from CLAP, AST, or WavLM, and optional calls to generative models such as Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑only controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
By Mengzhe Geng
arXiv:2610.02976v1 Announce Type: new
Abstract: Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about anoth...
By Hyunjae Ra, Aecheon Jung, Jungin Park, Sungeun Hong