arXiv:2606. 19381v1 Announce Type: cross Abstract: Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
Audio large language models (Audio LLMs) often fail to transcribe English‑Mandarin code‑switching speech, exhibiting language omission, translation‑instead‑of‑transcription, and hallucination. By applying Direct Preference Optimization (DPO) with 100 K preference pairs, the models learn to preserve mixed‑language content rather than translate, leading to significant reductions in mixed‑error rates (up to 89.6% in‑distribution). The study demonstrates that DPO can effectively align multilingual Audio LLMs for accurate code‑switching transcription.
The paper introduces a phoneme-guided text-to-speech (TTS) augmentation pipeline for automatic speech recognition (ASR) that links multilingual speech generation with candidate-text selection and reference-speech quality control. It proposes phoneme-frequency-guided selection (PFGS), which prioritizes candidate texts containing common phonetic content based on real ASR training transcripts. Experiments across four languages and 13 test sets show that random text selection improves recognition on 11 test sets, while PFGS further improves nine test sets with relative word error rate reductions up to 19.3%, and reference-speech filtering also contributes to performance gains.
The paper surveys how code‑mixed (CM) text and speech systems transfer knowledge between CM data and the native languages involved, a process that can go in both directions. It introduces a taxonomy of transfer direction, route, and modality, and finds that most work transfers from native languages to CM, while the reverse is newer and mainly aligns the two languages. The study notes that text and speech methods overlap but rarely cite each other, and that native data often outperforms new architectures on shared speech benchmarks, highlighting open questions about data mixing and evaluation across scripts.
SyRA-CS is a synthetic-to-real adaptation framework for code‑switched ASR that uses Reptile meta‑learning to create pair‑specific initializations from synthetic CS speech. For each language pair and pretrained model, the system meta‑trains on a synthetic CS‑FLEURS subset and then adapts with just one hour of labeled real target speech. Across four architectural settings and four language pairs, SyRA-CS consistently outperforms continual fine‑tuning, achieving up to 17.7% relative WER reduction for XLS‑R and 5.3–5.8% for other models.
The paper introduces a lightweight approach to detect code‑switching (CS) directly from speech by leveraging language‑selective hidden‑state dimensions in pretrained speech encoders. Using monolingual recordings to identify these dimensions, the method aggregates frame‑level activations to predict CS presence, achieving a mean F1 score of 0.798 on the CS‑FLEURS dataset with the Whisper‑small encoder. This demonstrates that pretrained speech models inherently encode CS information, offering a practical tool for expanding multilingual corpora.
Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding releva...