arXiv Computation and Language
Contextual Pictograms for Repairing Multilingual ASR Errors
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Aug 28
Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
The paper introduces a new benchmark called Speech-Augmented Visually Grounded Contrastive Triplet Benchmark, comprising 10,150 images from 18 MENA countries, each paired with a supported statement and two plausible but unsupported alternatives. It defines contrastive instability as the rate at which multimodal models fail to resolve all statements within a triplet, distinguishing fragmented reasoning from complete failure. Experiments on recent multimodal models show that shifts in modality (text vs. speech) and language (English vs. Arabic) lead to significant triplet-level inconsistencies, especially when speech is used, which are not fully reflected by overall accuracy metrics.
arXiv Computation and Language
Sep 25
Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systems
The paper introduces a cause-aware error recovery framework for cascaded Automatic Speech Recognition – Large Language Model (ASR‑LLM) pipelines in Spoken Dialogue Systems. It replaces simple ASR confidence filtering with precision‑focused detectors that use deep ASR latent representations to classify token‑level errors into perception, comprehension, and deletion failures. This fine‑grained diagnosis enables the LLM to execute targeted, multi‑turn clarification strategies, leading to a more than two‑fold increase in recall on domain‑shift errors and significant reductions in word error rate and downstream task errors across varied accents, distortions, and domains.
arXiv Computation and Language
Oct 21
Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors
arXiv:2610.24809v2 Announce Type: replace Abstract: Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during stre...
arXiv Computation and Language
Oct 12
Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR
The paper examines how different data‑augmentation techniques affect low‑resource automatic speech recognition beyond overall error rates. It compares speed perturbation, SpecAugment, and two text‑based audio synthesis methods across four low‑resource languages and English, analyzing phoneme‑level errors. The study finds that SpecAugment improves aggregate performance but increases tone and palatalization errors, while text‑based methods introduce consonant insertions; when combined, each method mitigates the other's weaknesses, yielding superadditive gains that diminish as data size grows to 100 hours.
arXiv AI
Jul 28
Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach
arXiv:2607. 23845v1 Announce Type: new Abstract: Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient signals a problem in speaking, hearing, or understanding, prompting the previous speaker to resolve it.
arXiv Computation and Language
Sep 18
Phoneme-guided TTS augmentation for ASR: A unified pipeline and multilingual evaluation
The paper introduces a phoneme-guided text-to-speech (TTS) augmentation pipeline for automatic speech recognition (ASR) that links multilingual speech generation with candidate-text selection and reference-speech quality control. It proposes phoneme-frequency-guided selection (PFGS), which prioritizes candidate texts containing common phonetic content based on real ASR training transcripts. Experiments across four languages and 13 test sets show that random text selection improves recognition on 11 test sets, while PFGS further improves nine test sets with relative word error rate reductions up to 19.3%, and reference-speech filtering also contributes to performance gains.