The paper introduces BASIS-Bench, a benchmark that separates demographic robustness, stereotypical bias, and personalization in speech large language models (LLMs). It evaluates eight recent speech LLMs, finding that while many models detect demographic cues, they rarely use them for appropriate personalization, and explicit demographic labels can worsen stereotypical associations. The study underscores the necessity of training speech LLMs to discern when demographic information should influence responses.
arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real hum...
The paper examines how large language models (LLMs) respond to different demographic cues—such as names—when users seek advice, focusing on race and gender in a U.S. context. It finds that using different cues for the same group leads to only partially overlapping changes in model responses, producing inconsistent conclusions about personalization and unstable bias metrics. The authors argue that LLMs react to linguistic signals tied to cues rather than to stable demographic categories, and they call for evaluations that use multiple cues and consider underlying mechanisms.
Large language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity. Previous work has documented this behavior, but the connection between these behavioral changes and the model's internal activations remains unclear.
DEAF (Diagnostic Evaluation of Acoustic Faithfulness) is a benchmark comprising over 2,700 conflict stimuli that test Audio Multimodal Large Language Models (Audio MLLMs) across three acoustic dimensions: emotional prosody, background sounds, and speaker identity. The benchmark uses a controlled multi-level evaluation framework that gradually increases textual influence, from semantic conflicts to misleading prompts, to separate content-driven bias from prompt-induced sycophancy. Diagnostic metrics are introduced to measure how much models rely on textual cues versus acoustic signals, and an evaluation of seven Audio MLLMs shows that while models respond to acoustic variations, their predictions are largely driven by textual inputs, highlighting a gap between benchmark performance and true acoustic understanding.
arXiv:2608. 11735v1 Announce Type: cross Abstract: Large language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity.
Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representations. We ask whether speaker-linked attributes that...
arXiv:2609.18533v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representation...
arXiv:2608.28833v1 Announce Type: new Abstract: While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providin...
arXiv:2608. 09930v1 Announce Type: cross Abstract: Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored.
The paper presents methods to reduce gender bias in audio‑enabled large language models (AuLLMs) by using adversarial training to unlearn gender perception without harming performance on other tasks. It also introduces a distillation approach that retains gender awareness but aligns responses with a gender‑agnostic teacher unless explicitly prompted. Experiments on three AuLLMs show promising results for gender‑unlearning and partial success for the distillation technique.
arXiv:2606. 06614v1 Announce Type: cross Abstract: Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data.