← Back to all news
arXiv AI By Hyebin Cho, Suho Yoo, Jihoo Jung, Joon Son Chung

Tracing Audio Grounding and Answer Selection in Audio LLMs

Read the original on arXiv AI →

The paper investigates how Audio Large Language Models (Audio LLMs) actually use audio input to determine answers, rather than relying on textual cues. It finds that replacing audio with silence or unrelated audio degrades performance more after training than before, that acoustic information shapes representations in early-to-middle layers and influences final predictions in middle-to-late layers, and that training impacts specific layer bands most strongly. These observations offer a mechanistic view of how training enhances the use of acoustic evidence in Audio LLMs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 3

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

AnchorPrompt is an adaptation technique for large audio‑language models that keeps the base model frozen and learns a single block of prompt vectors inserted at the decoder input. By training these prompts through self‑distillation on diverse audio and text perturbations, the method improves answer consistency and reduces hallucinations across multiple benchmarks. The approach is perturbation‑agnostic at inference, enabling zero‑shot transfer to unseen distortions such as reverberation and choice permutations.

By Pooneh Mousavi, Amir Ivry, Mirco Ravanelli, Cem Subakan
llmsragefficiencybenchmarkssafety
More like this →
arXiv Computation and Language
Sep 25

Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models

The paper introduces Reward‑Tilted On‑Policy Distillation (RT‑OPD), a method that enhances acoustic grounding in audio‑language models by using a frozen teacher to generate a reward based on the contrast between token predictions with and without audio. This reward reshapes the teacher distribution for reverse‑KL distillation, encouraging students to rely more on acoustic evidence. Experiments on two compact students across three benchmarks show that RT‑OPD consistently outperforms vanilla OPD, and a 3B model trained with RT‑OPD achieves 72.72% accuracy on the MMAU benchmark, surpassing other 3B models and rivaling larger 7B and 8B models.

By Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji
llmsefficiencybenchmarks
More like this →