arXiv AI
DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 28
When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue
The paper introduces ContraTalk, a benchmark that tests whether dialogue models truly use acoustic cues or rely on transcript shortcuts. It formalizes cross‑modal disagreement, creates conflict and consistent QA examples, and proposes an Audio Twin representation to expose acoustic evidence to models. Experiments show that while text‑only LLMs perform well on consistent cases, they falter on conflict cases, and AudioLLMs only partially mitigate this issue.
Hugging Face Trending Papers
Jun 23
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored.
arXiv Machine Learning
Jun 18
Investigating Faithfulness in Large Audio Language Models
arXiv:2509. 22363v4 Announce Type: replace Abstract: Large Audio Language Models (LALMs) integrate audio encoders with pretrained Large Language Models to perform complex multimodal reasoning tasks.
arXiv AI
Aug 11
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
arXiv:2608. 09930v1 Announce Type: cross Abstract: Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
arXiv Machine Learning
Sep 14
What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability
The paper introduces STAG, a post‑hoc framework that provides token‑level spectro‑temporal grounding for captions produced by audio‑based multimodal large language models (MLLMs). STAG estimates temporal support for each token via vocabulary projections of encoded audio, measures frequency‑band relevance through controlled spectral occlusion, and fuses these signals into a spectro‑temporal relevance map. Evaluations across ten explanation methods and four grounding benchmarks show that STAG achieves superior event‑localization performance on every dataset, and counterfactual deletion experiments confirm that removing the identified evidence selectively reduces model confidence and often eliminates the corresponding event from regenerated captions.
arXiv AI
Jun 11
Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents
arXiv:2606. 11219v1 Announce Type: cross Abstract: Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked.