Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored.
arXiv:2608. 09930v1 Announce Type: cross Abstract: Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
The paper investigates whether audio‑language models capture paralinguistic cues beyond spoken content. Using the Expresso dataset and four open‑source models, the authors trace how speaking style information is encoded in the late layers of the audio encoder but is degraded before reaching the final output. They find that some models are content‑driven while others are acoustic‑driven, revealing a gap between what is encoded and what is utilized in current audio‑language models.
Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paralinguistic evidence. We introduce counterfactual audits for paralinguistic response evaluation.
arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real hum...
arXiv:2605.28227v2 Announce Type: replace Abstract: Speech translation models are increasingly capable of preserving speech-specific information (e.g., speaker gender, prosody, and emphasis), yet eva...
arXiv:2607. 14846v1 Announce Type: cross Abstract: Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation.
arXiv:2606. 31729v1 Announce Type: cross Abstract: Text-to-speech (TTS) evaluation is an open challenge.
arXiv:2510. 25577v2 Announce Type: replace-cross Abstract: Recent advances in Speech Foundation Models (SFMs) enable direct processing of raw audio, allowing models to respond to subtle paralinguistic variation.
arXiv:2606. 11219v1 Announce Type: cross Abstract: Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked.
The paper introduces Live-ProsodyJudge (LPJ), a cost‑effective pairwise evaluator distilled from Gemini for assessing fine‑grained prosody in live streaming TTS. It identifies a flaw called verdict coupling, where multi‑dimensional scores collapse into a single preference, and proposes Decoupled‑Live‑ProsodyJudge (D‑LPJ) to eliminate this issue through masking and a span‑local GRPO strategy. Experiments show LPJ outperforms a single Gemini call in accuracy, and D‑LPJ provides independent dimension judgments, achieving high alignment with human top‑3 selections in a TTS candidate tournament.
arXiv:2608. 19936v1 Announce Type: cross Abstract: Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities.
arXiv:2608. 04479v1 Announce Type: cross Abstract: Text-to-audio (TTA) generation has recently achieved remarkable progress in synthesizing realistic audio from natural language descriptions.