The paper investigates a failure mode called phonological interference in phoneme‑level multilingual speech models, where models incorrectly impose a single language’s phonology on mixed or unseen language input. Experiments show significant loss of language‑specific phonemes in code‑switched and unseen‑language scenarios, and the authors trace this effect to a low‑dimensional subspace of internal activations. They propose a windowed language estimation (WLE) repair that reduces interference by 34–69% while preserving monolingual performance.
arXiv:2608. 01281v1 Announce Type: cross Abstract: Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling.
arXiv:2603. 29042v2 Announce Type: replace-cross Abstract: Phone recognition (PR) is a key enabler of multilingual and low-resource speech processing tasks, yet robust performance remains elusive.
The study evaluates phonologically informed tokenization for German speech recognition using a wav2vec 2.0 backbone fine‑tuned with CTC. Three tokenizer families—pretrained multilingual characters, data‑driven BPE, and phonologically informed units from syllabification and grapheme‑to‑phoneme conversion—are compared across 40 fine‑tunes on three German test sets (in‑domain read, dialectal spontaneous, and standard spontaneous speech). Results show that phonologically informed tokenizers match BPE and character baselines in‑domain, but under domain shift their performance depends on vocabulary size rather than linguistic variation, with syllable‑aware tokenization benefiting dialectal and spontaneous speech at small vocabularies; a phoneme‑level confusion analysis indicates that the acoustic encoder dominates error patterns.
We present DONDO, a family of open, permissively licensed automatic speech recognition (ASR) base models for African languages, built on the w2v-BERT 2. 0 self-supervised speech encoder.
The paper introduces task-informed parameter-efficient fine-tuning methods for low-resource speech recognition by applying Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR. Two extensions—Asymmetric-Coupled FCCA (AC‑FCCA) and Adaptive‑Rank FCCA (AR‑FCCA)—are proposed to exploit cross‑layer sharing and adapt rank allocation within a fixed parameter budget. Experiments on multilingual datasets show that standard FCCA matches or surpasses LoRA, while AR‑FCCA consistently improves performance across models without increasing trainable parameters.
arXiv:2608. 04586v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT).
arXiv:2608. 04586v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT).
arXiv:2607. 02633v1 Announce Type: new Abstract: We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling.
arXiv:2609. 11786v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized.
The study examines whether audio language models encode distinctive phonetic features similarly when processing spoken versus written input. By comparing mean representations of minimal phoneme pairs across six models, seven features, and 15 languages, the authors find that only voicing in the Qwen2.5-Omni models consistently shows a shared directional representation after correcting for multiple tests. The results suggest that the model family, rather than its size, determines how phonetic features are represented across audio and text streams.
arXiv:2601. 18904v3 Announce Type: replace-cross Abstract: Generative AI for speech and audio is increasingly expected to serve users across languages, cultures, and communities, yet current auditory Large Language Models (LLMs) are still largely trained and evaluated on high-resource data.
arXiv:2609.37121v1 Announce Type: new Abstract: Cross-linguistic effects are a central topic in bilingual first-language acquisition. Artificial learners can help investigate L1-L2 interactions by en...