arXiv Computation and Language

Marimba: A System One Model for Speech

arXiv AI
Oct 16

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

Qwen-Audio-Jev extends Qwen-Audio-3.1-ASR-Flash-Next with a dedicated decision head that unifies selection, binary judgment, and scoring into three primitives: CHOICE, NOUL, and SCORE. The model delivers competitive accuracies and error rates across 18 labeled configurations, achieving 78.70% on MMAU-mini, 84.06% on MMSU, 6.50% equal error rate on VoxCeleb1-O, and a 0.1062 normalized MAE on NISQA. Compared to its predecessor, Qwen-Audio-Jev offers roughly a 20‑fold inference speedup, enabling efficient high‑throughput audio decision‑making for tasks such as large‑scale screening, speaker verification, and automated quality assessment.

By Ying Fang, Zhifu Gao, Qing Gu, Jiajun He, Kangqi Jing, Xiangang Li, Yu Pu, Wupeng Wang
Hugging Face Trending Papers
4d ago

Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR

The paper presents a taxonomy of errors in streaming dictation ASR systems, moving beyond traditional word error rate (WER) metrics. It evaluates eleven ASR systems on a diverse test set of 106 scripted English dictations, 20 clips each in eleven other languages/dialects, and 400 bilingual recordings, categorizing errors into recognition, output formatting, language handling, and audio drop‑out. The study finds that systems with similar WER can fail in markedly different ways—e.g., some miss a large proportion of whispered words or fail to capture non‑English content—highlighting the importance of error type analysis for real‑time dictation. whyItMatters":"The findings reveal that conventional WER alone is insufficient for assessing real‑time dictation performance, as it masks critical failure modes that directly affect user experience."

arXiv Computation and Language
Oct 12

People Making Conversation: Evaluating How Naturally Speech Models Converse

The study evaluates conversational speech models by placing them at 427 pauses and 482 turn ends from 176 unscripted conversations, scoring their responses on timing, latency, length, wording, and recall of recent dialogue. Hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause, with latency ranging from 1–4 s for silence-based endpointers and about 8 s for semantic turn detection, while full‑duplex models (PersonaPlex, Moshi) remain silent more often and can interrupt within human reaction time. Recall rates vary: hosted systems remember 41–60 % of facts under a Claude Opus 5.5 judge and 19–26 % with a strict word matcher; full‑duplex models recall 17–22 % and 12–15 % respectively.

By Arya Bastani, Iman Modarressi
arXiv Computation and Language
Sep 24

Text Scores Can Miss Waveform Use: A Qwen2-Audio Quantization Case Study

The paper presents a new evaluation protocol for post‑training quantization of speech language models that separates lexical output, transcript‑insufficient endpoints, and packed implementations. In a Qwen2‑Audio case study, a 6‑bit allocation selected for translation improves chrF scores but degrades emotion recognition, while uniform and front‑layer controls perform better on emotion tasks. Similar patterns hold at 7 bits, and a 4‑bit study shows consistent emotion deficits across all low‑bit allocations, with no advantage for the selected scheme. The study highlights a precision‑dependent mismatch between lexical output, waveform‑dependent behavior, and nominal precision, without claiming a general failure of low‑bit models or a deployment benefit for the selected allocation.

By Mengzhe Geng, Jinxi Jin, Junhao Xu
arXiv Computation and Language
Oct 7

Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study

The study evaluates Qwen2‑Audio‑7B‑Instruct quantization at mixed 4/8‑bit settings on English‑to‑German translation and emotion recognition tasks. BLEU and chrF scores relative to FP16 show no significant difference, yet absolute accuracy on RAVDESS drops by 3.71% for 6‑bit and 1.17% for 7‑bit, with the 6‑bit interval statistically significant. The results demonstrate that text‑output metrics alone cannot confirm performance preservation on speech tasks where labels cannot be inferred from transcripts.

By Mengzhe Geng, Jinxi Ji, Junhao Xu
arXiv Computation and Language
6d ago

Earlybird: Predictive Streaming Speech Recognition in 29M Parameters

Earlybird is a 29‑million‑parameter streaming speech recognizer that predicts words before the speaker finishes them, achieving median correct predictions 90 ms early on clean audiobook speech. It learns from audio cut off at random points, including mid‑word, and outputs confidence scores for every word, including the one still being spoken. Across six diverse test sets—audible books, TED talks, public speech, earnings calls, and synthetic speech—Earlybird consistently outperforms existing streaming models, correctly predicting 69 % of words before they finish on audiobooks, compared to 3–25 % for other models.

By Sahil Mahendrakar
arXiv Machine Learning
Aug 28

Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores

The paper presents an interpretable, fair, and accurately benchmarked automated system for assessing second‑language English speaking. Using a hybrid of feature‑based speech‑timing metrics and a large language model (LLM) fluency judgment, the system achieves a Spearman correlation of 0.818 with the ICNALE Global Rating Archive, outperforming 81 % of trained human raters. A controlled study shows that encoding pauses into the LLM prompt does not meaningfully affect fluency scores, indicating that the system’s fluency signal derives from measurable speech‑timing features.

By Eichi Uehara
arXiv Computation and Language
Aug 31

Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluation

The paper investigates how to evaluate audio‑language models by separating the use of acoustic evidence from the need to invoke a generative audio model. Using a controlled call‑decision framework, the authors compare policies that rely on transcript labels, encoder outputs from CLAP, AST, or WavLM, and optional calls to generative models such as Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑only controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).

By Mengzhe Geng
arXiv Computation and Language
Oct 22

How Far Can the Gemma 4 Speech Encoder Go Without Changing Its Weights? Saudi and Gulf Arabic Speech Recognition with a Frozen and an Adapter-Augmented Encoder

The paper evaluates a 305 M‑parameter audio encoder shared by Gemma 4 and EmbeddingGemma 2 for Saudi and Gulf Arabic speech recognition without altering its weights. Two approaches are compared: a frozen system that trains only a 22.6 M‑parameter CTC reader, and an adapted system that adds 3.1 M low‑rank adapters, a 41.5 M‑parameter hybrid CTC/attention reader, and joint decoding with a sub‑word language model. The adapted system matches or surpasses a 0.6 B‑parameter fine‑tuned recogniser on several datasets, while the frozen system remains competitive in character error rate and runs three‑quarters faster on CPU with greedy decoding.

By Amr Abdelkhalik
arXiv Computation and Language
Sep 15

Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluatio

The paper investigates how to evaluate generative audio large language models (Audio‑LLMs) on known closed‑set tasks by separating the decision to call a generative model from the use of acoustic evidence. It introduces a controlled call‑decision framework where a policy can choose between a transcript label, encoder evidence from CLAP, AST, or WavLM, or a generative call to Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio, and measures the impact of generative calls on accuracy. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑based controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).

By Mengzhe Geng