arXiv:2609.13150v1 Announce Type: cross Abstract: Reference-free quality predictors such as UTMOS, DNSMOS and SCOREQ are the de facto automatic evaluators for text-to-speech (TTS) and are increasingl...
arXiv:2607. 21820v1 Announce Type: cross Abstract: Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks.
The paper investigates how to evaluate generative audio large language models (Audio‑LLMs) on known closed‑set tasks by separating the decision to call a generative model from the use of acoustic evidence. It introduces a controlled call‑decision framework where a policy can choose between a transcript label, encoder evidence from CLAP, AST, or WavLM, or a generative call to Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio, and measures the impact of generative calls on accuracy. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑based controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
The paper investigates how to evaluate audio‑language models by separating the use of acoustic evidence from the need to invoke a generative audio model. Using a controlled call‑decision framework, the authors compare policies that rely on transcript labels, encoder outputs from CLAP, AST, or WavLM, and optional calls to generative models such as Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑only controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
SpeechCritic learns a diagnostic speech judge from only about 300 human-labeled comparisons, calibrating a frontier audio-language model with non-binding hints that select acoustic measurements agreeing with human judgments. The approach improves dimension-level agreement by 6.3 points and reduces mismatch in Tie rates by 10.4 points, and training a 7B judge with SFT, OPD, and RL yields distinct behaviors, with RL producing more specific rationales. The method works across language pairs, demonstrating a path from limited human preferences to a diagnostic speech judge.
Qwen-Audio-Jev extends Qwen-Audio-3.1-ASR-Flash-Next with a dedicated decision head that unifies selection, binary judgment, and scoring into three primitives: CHOICE, NOUL, and SCORE. The model delivers competitive accuracies and error rates across 18 labeled configurations, achieving 78.70% on MMAU-mini, 84.06% on MMSU, 6.50% equal error rate on VoxCeleb1-O, and a 0.1062 normalized MAE on NISQA. Compared to its predecessor, Qwen-Audio-Jev offers roughly a 20‑fold inference speedup, enabling efficient high‑throughput audio decision‑making for tasks such as large‑scale screening, speaker verification, and automated quality assessment.
arXiv:2609.33212v2 Announce Type: replace Abstract: Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems...
Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paralinguistic evidence. We introduce counterfactual audits for paralinguistic response evaluation.
The paper presents an interpretable, fair, and accurately benchmarked automated system for assessing second‑language English speaking. Using a hybrid of feature‑based speech‑timing metrics and a large language model (LLM) fluency judgment, the system achieves a Spearman correlation of 0.818 with the ICNALE Global Rating Archive, outperforming 81 % of trained human raters. A controlled study shows that encoding pauses into the LLM prompt does not meaningfully affect fluency scores, indicating that the system’s fluency signal derives from measurable speech‑timing features.
arXiv:2610.16826v1 Announce Type: cross Abstract: Deepfake speech detectors are typically evaluated as binary classifiers on fixed benchmarks, but deployed systems face evolving attacks, codecs, spea...
The paper introduces SITAR, a situational distortion metric for evaluating automatic speech recognition (ASR) systems. SITAR aggregates situational severity and cumulative extent into a unified score using a bounded recursive formulation, and is implemented as a lightweight BERT-family regressor that requires minimal runtime cost. The authors validate SITAR through extrinsic and intrinsic protocols, showing it tracks capability scaling and provides diagnostic utility across diverse datasets and models.
arXiv:2606. 10912v1 Announce Type: cross Abstract: Deepfake speech detectors often output a single score without explaining why an audio sample is flagged, where in the signal the evidence lies, or what cues drive the decision.