The paper investigates how to evaluate audio‑language models by separating the use of acoustic evidence from the need to invoke a generative audio model. Using a controlled call‑decision framework, the authors compare policies that rely on transcript labels, encoder outputs from CLAP, AST, or WavLM, and optional calls to generative models such as Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑only controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
By Mengzhe Geng
VoxReason introduces a listener‑free evaluation framework that measures whether a speech planning system’s delivery choices—such as pitch, energy, rate, pause, emphasis, and stance—are grounded in cited source records before any waveform is generated. The system outputs a source‑cited speaking plan and uses a deterministic verifier to check citation legality, slot agreement, unsupported states, schema validity, and counterfactual locality. Experiments on 1,440 source‑label cases show that simple slot accuracy can be misleading, while a 7B locality‑based repair model significantly improves plan‑slot accuracy and locality, and removing source records sharply reduces the grounded score.
whyItMatters":"The framework provides a concrete, measurable way to ensure that expressive speech systems make source‑licensed planning decisions, addressing a source‑use failure that occurs before audio synthesis."
By Mengzhe Geng
arXiv:2608. 19515v1 Announce Type: new Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged.
By Xinyi Liu, Hooshang Nayyeri, Dilek Hakkani-Tur, Emine Yilmaz, JK Kim, Yifei Zhang, Charith Peris, Hari Thadakamalla
arXiv:2609.08899v2 Announce Type: replace-cross
Abstract: Speech deepfakes can mimic a speaker's voice convincingly enough to deceive listeners and automated systems. This has driven strong progress...
By Mengzhe Geng, Yujia Lu, Patrick Littell, Manuela Kunz, Xie Chen
arXiv:2608.27783v3 Announce Type: replace-cross
Abstract: Speech language models (speech LLMs) can generate plausible outputs from audio that contains no usable speech evidence. We study this failure...
By Mengzhe Geng
Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions.
AVERT is a method for spoken dialogue state tracking that improves upon a per-turn text editor by incorporating an audio-conditioned verifier to score candidate slot values. It addresses three types of recoverable errors—inconsistent values across turns, omitted slots, and values unsupported by audio—using three specialized operators: vote, add, and swap, each limited to relevant slots. On the SpokenWOZ dataset, AVERT achieves a joint goal accuracy of 40.13, surpassing both a base speech-LLM (33.04) and a text editor (38.34) without retraining, and matching the performance of a larger end‑to‑end system that processes the full spoken history.
By Chunggi Lee, Hanspeter Pfister
arXiv:2609.05871v1 Announce Type: cross
Abstract: Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-s...
By Song-ha Jo, Sehyun Lee, Soyoon Kim, Jaesik Choi, Sanghyuk Choi
The paper introduces the Speech-Unsupported Rejection Evaluation Challenge (SURE‑Challenge), a benchmark designed to test whether speech‑LLMs should accept or reject audio inputs before generating answers. Using LibriSpeech‑derived transcriptions paired with first‑word question answering, the authors evaluate various noise and silence conditions, and compare a simple energy‑plus‑Whisper‑score rule against a Qwen2‑Audio front‑end. On a 474‑row test set, the rule rejects 196 of 204 unsupported inputs while preserving accuracy on supported data, revealing a pre‑generation error mode that answer‑only scoring misses.
By Mengzhe Geng
arXiv:2608.28916v1 Announce Type: new
Abstract: Automatic speech recognition (ASR) systems are commonly evaluated with word error rate (WER), yet many voice workflows depend on exact written values f...
By Tyler Baumgartner, Brandon Tai, Lisa Kaelin-Martin, Candice Fan, Luc Debaupte, Bill Wang, Yi Zhong
TamilEOT is a new dataset and two audio‑only models for detecting semantic end‑of‑turn in Tamil telephone speech, comprising 18,485 labeled turn boundaries from 116 real conversations. The models, fine‑tuned from Smart Turn v3, achieve 83.71% and 86.13% accuracy on a held‑out set, with ROC‑AUC rising from 0.751 to 0.921, and run in under 150 ms on a laptop CPU. The paper also reports the cost of building the dataset, the accuracy of rule‑derived labels versus human agreement, and the impact of encoder capacity on performance.
whyItMatters":"The study provides the first publicly available Tamil end‑of‑turn detection resource, enabling more accurate voice agents for South Indian languages and demonstrating efficient, low‑latency models that outperform zero‑shot baselines."
By Santhoshkumar V
The study examined whether providing full prompt-level context to a large multimodal model would improve speech transcription accuracy on a production oral‑history corpus. Using a preregistered within‑item paired ablation, the authors found that adding context did not produce a detectable change in side‑level word error rate (WER) for either gpt‑4o‑transcribe or gemini‑2.5‑flash. The results suggest that context alone may not be sufficient to enhance aggregate transcription accuracy, and that finer‑grained, sequence‑aligned metrics are needed to evaluate such mechanisms.
By Theodore O. Cochran, Stephanie Dodson, Keith Nore