The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2609.13150v1 Announce Type: cross Abstract: Reference-free quality predictors such as UTMOS, DNSMOS and SCOREQ are the de facto automatic evaluators for text-to-speech (TTS) and are increasingl...
arXiv:2607. 21820v1 Announce Type: cross Abstract: Audio deepfake detectors are trained to distinguish genuine speech from synthetic speech and often perform well on standard benchmarks.
The paper investigates how to evaluate generative audio large language models (Audio‑LLMs) on known closed‑set tasks by separating the decision to call a generative model from the use of acoustic evidence. It introduces a controlled call‑decision framework where a policy can choose between a transcript label, encoder evidence from CLAP, AST, or WavLM, or a generative call to Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio, and measures the impact of generative calls on accuracy. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑based controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
The paper investigates how to evaluate audio‑language models by separating the use of acoustic evidence from the need to invoke a generative audio model. Using a controlled call‑decision framework, the authors compare policies that rely on transcript labels, encoder outputs from CLAP, AST, or WavLM, and optional calls to generative models such as Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑only controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
SpeechCritic learns a diagnostic speech judge from only about 300 human-labeled comparisons, calibrating a frontier audio-language model with non-binding hints that select acoustic measurements agreeing with human judgments. The approach improves dimension-level agreement by 6.3 points and reduces mismatch in Tie rates by 10.4 points, and training a 7B judge with SFT, OPD, and RL yields distinct behaviors, with RL producing more specific rationales. The method works across language pairs, demonstrating a path from limited human preferences to a diagnostic speech judge.
Qwen-Audio-Jev extends Qwen-Audio-3.1-ASR-Flash-Next with a dedicated decision head that unifies selection, binary judgment, and scoring into three primitives: CHOICE, NOUL, and SCORE. The model delivers competitive accuracies and error rates across 18 labeled configurations, achieving 78.70% on MMAU-mini, 84.06% on MMSU, 6.50% equal error rate on VoxCeleb1-O, and a 0.1062 normalized MAE on NISQA. Compared to its predecessor, Qwen-Audio-Jev offers roughly a 20‑fold inference speedup, enabling efficient high‑throughput audio decision‑making for tasks such as large‑scale screening, speaker verification, and automated quality assessment.