Audio large language models (Audio LLMs) often fail to transcribe English‑Mandarin code‑switching speech, exhibiting language omission, translation‑instead‑of‑transcription, and hallucination. By applying Direct Preference Optimization (DPO) with 100 K preference pairs, the models learn to preserve mixed‑language content rather than translate, leading to significant reductions in mixed‑error rates (up to 89.6% in‑distribution). The study demonstrates that DPO can effectively align multilingual Audio LLMs for accurate code‑switching transcription.
By Trung Nguyen Quang, Cheng Yi Lewis Won, Minh Duc Pham, Yingxu He, Shuo Sun, Ai Ti Aw
The paper introduces a phoneme-guided text-to-speech (TTS) augmentation pipeline for automatic speech recognition (ASR) that links multilingual speech generation with candidate-text selection and reference-speech quality control. It proposes phoneme-frequency-guided selection (PFGS), which prioritizes candidate texts containing common phonetic content based on real ASR training transcripts. Experiments across four languages and 13 test sets show that random text selection improves recognition on 11 test sets, while PFGS further improves nine test sets with relative word error rate reductions up to 19.3%, and reference-speech filtering also contributes to performance gains.
By Zhen Wang, TianRui Wu, RongQi Han, Hao Wu, Wei Liang, Wei Xu
The paper surveys how code‑mixed (CM) text and speech systems transfer knowledge between CM data and the native languages involved, a process that can go in both directions. It introduces a taxonomy of transfer direction, route, and modality, and finds that most work transfers from native languages to CM, while the reverse is newer and mainly aligns the two languages. The study notes that text and speech methods overlap but rarely cite each other, and that native data often outperforms new architectures on shared speech benchmarks, highlighting open questions about data mixing and evaluation across scripts.
By Debajyoti Mazumder, Akash Anil
SyRA-CS is a synthetic-to-real adaptation framework for code‑switched ASR that uses Reptile meta‑learning to create pair‑specific initializations from synthetic CS speech. For each language pair and pretrained model, the system meta‑trains on a synthetic CS‑FLEURS subset and then adapts with just one hour of labeled real target speech. Across four architectural settings and four language pairs, SyRA-CS consistently outperforms continual fine‑tuning, achieving up to 17.7% relative WER reduction for XLS‑R and 5.3–5.8% for other models.
By Fadel Thior, Ahmer Tabassum, Haania Siddiqui, Hanan Aldarmaki, Thamar Solorio
The paper introduces a lightweight approach to detect code‑switching (CS) directly from speech by leveraging language‑selective hidden‑state dimensions in pretrained speech encoders. Using monolingual recordings to identify these dimensions, the method aggregates frame‑level activations to predict CS presence, achieving a mean F1 score of 0.798 on the CS‑FLEURS dataset with the Whisper‑small encoder. This demonstrates that pretrained speech models inherently encode CS information, offering a practical tool for expanding multilingual corpora.
By Jie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel, Natalie Schluter
Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding releva...
The paper introduces a unified phoneme‑based TTS‑to‑ASR augmentation pipeline that uses a multilingual TTS model with language‑ID conditioning and incorporates grapheme‑to‑phoneme conversion, reference‑speech filtering, and candidate‑text selection. It proposes phoneme‑frequency‑guided selection (PFGS) to rank sentences based on phoneme frequencies from real ASR labels, and demonstrates that random augmentation and PFGS both improve ASR performance across Arabic, French, Italian, and Portuguese test sets, with PFGS yielding up to a 19.3% relative WER reduction. The study also shows that filtering reference speech can further lower WER by up to 0.59 points on certain datasets.
By Zhen Wang, TianRui Wu, RongQi Han, Hao Wu, Wei Liang
arXiv:2609.01016v1 Announce Type: new
Abstract: Current speech synthesis struggles with code-switching, which mixes a foreign language phrase into a primary language utterance, causing the phrase to...
By Che Hyun Lee, Sangkwon Park, Donghun Kang, Dongwook Lee, Youngho Cho, Heeseung Kim, Sungroh Yoon
The paper addresses the challenge of identifying code‑switched utterances in language identification systems. It revisits the MaskLID approach, highlighting its overreliance on word‑level language association scores, and reformulates its optimization as an Integer Linear Program to incorporate clear, interpretable constraints. These enhancements significantly improve performance across ten diverse languages on code‑switching benchmarks, with the authors releasing code and data for reproducibility.
By Joanna Rado{\l}a, Josep Maria Crego, Fran\c{c}ois Yvon
The paper introduces a new benchmark of 13,044 spontaneous Nepali‑English code‑switched podcast utterances, totaling 15.45 h, and evaluates a two‑stage synthetic‑to‑real adaptation framework for ASR. Stage 1 uses supervised fine‑tuning on 0.95 h of synthetic speech, reducing Mixed Error Rate from 23.40 % to 21.52 % and switch‑point error rate from 35.89 % to 28.77 %. Stage 2 applies Group Relative Policy Optimization with script and transcription rewards, yielding modest but statistically significant further improvements, particularly in English script confusion and switch‑point accuracy. The study also shows that an auxiliary script reward term offers no benefit beyond MER‑only optimization in this setting, and that the synthetic SFT provides the main cross‑domain transfer to spontaneous human speech.
By Firoj Paudel
arXiv:2604.06487v2 Announce Type: replace
Abstract: Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR arch...
By Thibault Ba\~neras-Roux, Sergio Burdisso, Esa\'u Villatoro-Tello, Dairazalia S\'anchez-Cort\'es, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke
AfriSwitch is a 61.36‑hour, human‑transcribed benchmark of in‑the‑wild code‑switched speech covering 16 African languages and varieties, annotated with switch‑level English span tags, per‑utterance Code‑Mixing Index (CMI), and switch‑point counts. The corpus reveals that code‑switching behaviour varies widely across languages, with no single metric fully capturing how code‑switched a language is. Benchmarking five open and commercial multilingual ASR systems in a zero‑shot setting shows high word error rates, with the best system averaging 35.93% WER and none dropping below 24% on any language, indicating that Africa‑targeted training rather than model scale or nominal language coverage best predicts performance.
By Gabrial Zencha Ashungafac, Busayo Awobade, Tobi Olatunji