Measuring benchmark optimization in speech recognition

Towards Quantifying Benchmark Optimization in ASR Models

arXiv:2608. 19936v1 Announce Type: cross Abstract: Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities.

TTS Arena: Benchmarking Text-to-Speech Models in the Wild

RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

arXiv:2607. 14846v1 Announce Type: cross Abstract: Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation.

Performance and Complexity Trade-off Optimization of Speech Models During Training

The paper introduces a reparameterization technique that injects feature noise to jointly optimize speech model performance and computational complexity during training. Unlike traditional pruning, this method dynamically adjusts model size for a desired performance‑complexity trade‑off without heuristic weight removal. The authors validate their approach with a synthetic example and two real‑world applications—voice activity detection and audio anti‑spoofing—providing publicly available code for further research.

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

arXiv:2605. 13087v2 Announce Type: replace-cross Abstract: Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance.

Benchmarking Automatic Speech Recognition Tools for Iberian Languages

arXiv:2609.36920v1 Announce Type: new Abstract: Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficienc...

VOSSA: Voiceprint Optimization for Streaming Speech Architectures

arXiv:2609.38887v1 Announce Type: cross Abstract: Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effect...

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

DiscoPhon is a multilingual benchmark designed to evaluate unsupervised phoneme discovery from discrete speech units. It includes 6 development and 6 test languages that cover a wide range of phonemic contrasts, and requires systems to generate discrete units mapped to a predefined phoneme inventory using only 10 hours of speech from an unseen language. The benchmark assesses unit quality, recognition, and segmentation, and provides four pretrained multilingual HuBERT and SpidR baselines that demonstrate current models can produce units that correlate well with phonemes, though performance varies across languages.

Scaling Audio Models Efficiently: A Joint Study of Compute Constraints and Optimization Behavior

arXiv:2606. 22790v2 Announce Type: replace-cross Abstract: In this paper, we investigate the tradeoffs between compute allocation and model performance for two speech processing tasks: Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER).

Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR

The paper examines how different data‑augmentation techniques affect low‑resource automatic speech recognition beyond overall error rates. It compares speed perturbation, SpecAugment, and two text‑based audio synthesis methods across four low‑resource languages and English, analyzing phoneme‑level errors. The study finds that SpecAugment improves aggregate performance but increases tone and palatalization errors, while text‑based methods introduce consonant insertions; when combined, each method mitigates the other's weaknesses, yielding superadditive gains that diminish as data size grows to 100 hours.

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored.

Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsity

The paper introduces a compression framework for the Whisper automatic speech recognition model that jointly optimizes six deployment dimensions—model size, temporal resolution, encoder token stride, low‑rank adaptation capacity, weight precision, and sparsity pattern—using NSGA‑III. The optimization targets three objectives: word error rate, inference FLOPs, and memory footprint. Evaluating 1,680 configurations, the study identifies compression combinations that outperform single‑axis scaling and notes that 1:4 structured sparsity cannot maintain acceptable accuracy within the tested budgets.