← Back to all news
arXiv Computation and Language By Kirill Borodin, Vasilii Kudryavtsev, Nikita Vasiliev, Grach Mkrtchian

Measuring the Verifier's Floor: How Much Apparent Rare-Word Text-to-Speech Error Is ASR Error?

Read the original on arXiv Computation and Language →

The paper investigates how much of the apparent error in text-to-speech (TTS) systems is actually due to automatic speech recognition (ASR) errors. By transcribing human recordings of test sentences, the authors find that ASR systems miss 33–67% of very rare words on LibriSpeech and FLEURS, and all three recognizers miss 22–42% of such words. Using multiple recognizer families reduces the apparent TTS error on rare words but still leaves them as the hardest to evaluate, indicating that ignoring ASR misses biases the assessment and that a listening test is required.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
21h ago

Fine-Tuning a Transducer for Impaired Speech: Where the Gains Appear and What They Mean for Evaluation

The paper reports on fine‑tuning a transducer‑based speech recognizer for impaired speech, achieving a 7.81% character error rate on a hidden test set—well below the 15.26% baseline. The fine‑tuned model ran on CPUs with a median latency of ~0.9 s and performed best on speakers that were hardest for the off‑the‑shelf recognizer. The authors note that fine‑tuning yields larger gains on new speakers and on unseen sentences, and they discuss implications for evaluating such systems.

By Kien Nguyen, Tuan Dinh, Bao Phan
fine-tuningmultimodal
More like this →
arXiv AI
Sep 15

Building a Production Greek-English Speech Recognizer

The paper details a multi‑month engineering effort to build Sophea, a production Greek‑English automatic speech recognition system. It evaluates the system against nine production gates—including word error rate, language identification, and hallucinations—across twenty‑three training iterations and two model architectures, finding that no single training‑data composition satisfies all gates. The authors describe a six‑stage data pipeline that dramatically reduces discarded Greek audio, isolate a hallucination defect to a specific training package, and demonstrate that a three‑model ROVER ensemble achieves full gate coverage and a 29 % relative WER improvement. A separate learned arbiter model is released on the Open ASR Leaderboard with 4.26 % average WER on English tests and 25.88 % on live Greek noisy traffic.

By Christos Petrocheilos, Cleopatra Papadopoulou, Chris Porikis, Ioakeim Perros, Ayoub Kirouane, Themistoklis Nikolis
multimodalbenchmarkssafetydata-engineering
More like this →
arXiv Computation and Language
Oct 9

FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech

FA-Bench is an open framework that standardizes forced alignment evaluation by fixing transcript normalization, data splits, phone mapping, and scoring procedures. It offers two tracks: one with reference transcripts and another with recognizer outputs, evaluated on clean and four degraded audio conditions using 21 open models and 9 commercial APIs. The benchmark scores every utterance boundary with a tolerance‑based F1 metric, revealing systematic timing biases such as Whisper’s 150 ms early alignment and commercial APIs’ 50 ms late alignment.

By Wei Chu, Yuanzhe Dong, Ke Tan, Dong Han, Yichao Zhou, Ruchao Fan, Bingshen Mu, Jingbei Li, Vishwas Shetty, Sarthak Bisht, Ziyue Qiu, Massa Baali, Rita Singh, Bhisha Raj
multimodalbenchmarkssafety
More like this →