arXiv:2609.35952v1 Announce Type: cross
Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real hum...
By Shen Yan, Duc Le, Irina-Elena Veliche
The paper examines how large language models (LLMs) respond to different demographic cues—such as names—when users seek advice, focusing on race and gender in a U.S. context. It finds that using different cues for the same group leads to only partially overlapping changes in model responses, producing inconsistent conclusions about personalization and unstable bias metrics. The authors argue that LLMs react to linguistic signals tied to cues rather than to stable demographic categories, and they call for evaluations that use multiple cues and consider underlying mechanisms.
By Manuel Tonneau, Neil K. R. Sehgal, Niyati Malhotra, Sharif Kazemi, Victor Orozco-Olvera, Ana Mar\'ia Mu\~noz Boudet, Lakshmi Subramanian, Samuel P. Fraiberger, Sharath Chandra Guntuku, Valentin Hofmann
Large language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity. Previous work has documented this behavior, but the connection between these behavioral changes and the model's internal activations remains unclear.
DEAF (Diagnostic Evaluation of Acoustic Faithfulness) is a benchmark comprising over 2,700 conflict stimuli that test Audio Multimodal Large Language Models (Audio MLLMs) across three acoustic dimensions: emotional prosody, background sounds, and speaker identity. The benchmark uses a controlled multi-level evaluation framework that gradually increases textual influence, from semantic conflicts to misleading prompts, to separate content-driven bias from prompt-induced sycophancy. Diagnostic metrics are introduced to measure how much models rely on textual cues versus acoustic signals, and an evaluation of seven Audio MLLMs shows that while models respond to acoustic variations, their predictions are largely driven by textual inputs, highlighting a gap between benchmark performance and true acoustic understanding.
By Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Yutong Zhang, Ziteng Wang, Ruofan Liao, Weisheng Xu, Sichen Liu
arXiv:2608. 11735v1 Announce Type: cross Abstract: Large language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity.
By Yueru Yan, Siqi Wu, Thai Le
Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representations. We ask whether speaker-linked attributes that...