arXiv Computation and Language

Phoneme-guided TTS augmentation for ASR: A unified pipeline and multilingual evaluation

The paper introduces a phoneme-guided text-to-speech (TTS) augmentation pipeline for automatic speech recognition (ASR) that links multilingual speech generation with candidate-text selection and reference-speech quality control. It proposes phoneme-frequency-guided selection (PFGS), which prioritizes candidate texts containing common phonetic content based on real ASR training transcripts. Experiments across four languages and 13 test sets show that random text selection improves recognition on 11 test sets, while PFGS further improves nine test sets with relative word error rate reductions up to 19.3%, and reference-speech filtering also contributes to performance gains.

arXiv Computation and Language
Aug 28

Scaling phoneme-based TTS augmentation for ASR: A unified pipeline and controlled study

The paper introduces a unified phoneme‑based TTS‑to‑ASR augmentation pipeline that uses a multilingual TTS model with language‑ID conditioning and incorporates grapheme‑to‑phoneme conversion, reference‑speech filtering, and candidate‑text selection. It proposes phoneme‑frequency‑guided selection (PFGS) to rank sentences based on phoneme frequencies from real ASR labels, and demonstrates that random augmentation and PFGS both improve ASR performance across Arabic, French, Italian, and Portuguese test sets, with PFGS yielding up to a 19.3% relative WER reduction. The study also shows that filtering reference speech can further lower WER by up to 0.59 points on certain datasets.

By Zhen Wang, TianRui Wu, RongQi Han, Hao Wu, Wei Liang
arXiv Computation and Language
Oct 12

Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR

The paper examines how different data‑augmentation techniques affect low‑resource automatic speech recognition beyond overall error rates. It compares speed perturbation, SpecAugment, and two text‑based audio synthesis methods across four low‑resource languages and English, analyzing phoneme‑level errors. The study finds that SpecAugment improves aggregate performance but increases tone and palatalization errors, while text‑based methods introduce consonant insertions; when combined, each method mitigates the other's weaknesses, yielding superadditive gains that diminish as data size grows to 100 hours.

By Katsumi Ibaraki, David Chiang
Hugging Face Trending Papers
Jun 2

Efficient ASR Training with Conversations that Never Happened

Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations.

arXiv Computation and Language
Oct 8

Phoneme-Guided Initialization for LLM-based Speech Recognition

The paper introduces phoneme‑guided initialization for large language model‑based speech recognition. By pre‑training an audio encoder on speech‑to‑phoneme conversion and an LLM on phoneme‑to‑grapheme conversion, then fine‑tuning jointly on ASR, the method leverages phoneme‑mediated processing within an end‑to‑end framework. Experiments on Japanese, Chinese, Tatar, and Urdu demonstrate that this approach matches or surpasses both cascaded S2P‑P2G baselines and end‑to‑end models without phoneme initialization.

By Ryo Magoshi, Shinsuke Sakai, Tatsuya Kawahara
arXiv Computation and Language
Sep 22

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

arXiv:2604.06487v2 Announce Type: replace Abstract: Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR arch...

By Thibault Ba\~neras-Roux, Sergio Burdisso, Esa\'u Villatoro-Tello, Dairazalia S\'anchez-Cort\'es, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke
arXiv Computation and Language
Oct 28

Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization

The paper introduces a linguistic-aware agentic data sampling framework for multilingual automatic speech recognition (ASR). It embeds a data sampling LLM agent into the training loop, guided by a rubric that analyzes ASR errors across phonological, morphological, and syntactic dimensions. Experiments on twelve Common Voice languages with Whisper-medium demonstrate that the method outperforms full-data training while using less data, fewer computational resources, and achieving faster convergence.

By Zhiheng Qian, Junhao Du, Jing Peng, Hanqi Li, Kunyang Peng, Feng Xue, Lu Chen, Shuai Wang, Kai Yu