← Back to all news
arXiv AI By Theodore O. Cochran, Stephanie Dodson, Keith Nore

No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus

Read the original on arXiv AI →

The study examined whether providing full prompt-level context to a large multimodal model would improve speech transcription accuracy on a production oral‑history corpus. Using a preregistered within‑item paired ablation, the authors found that adding context did not produce a detectable change in side‑level word error rate (WER) for either gpt‑4o‑transcribe or gemini‑2.5‑flash. The results suggest that context alone may not be sufficient to enhance aggregate transcription accuracy, and that finer‑grained, sequence‑aligned metrics are needed to evaluate such mechanisms.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 19

ASR-Style Transcript Normalization Removes Useful Pragmatic Signal for Dialogue Act Classification

The study investigates how automatic speech recognition (ASR) transcript normalization—specifically the removal of filled pauses, punctuation, and colloquial expansions—affects dialogue act classification. By testing nine transcript conditions across six classifiers and three conversational corpora, the authors find that heavy normalization consistently lowers Macro‑F1 scores, especially when using a comprehensive filler lexicon. Punctuation removal and selective filler removal also degrade performance, indicating that the impact of normalization varies with preprocessing choices, feature extraction methods, and the conversational context.

By Lucky Onyekwelu-Udoka, Md Shafiqul Islam
llmsnlpmultimodal
More like this →
arXiv Computation and Language
6d ago

Synthetic-to-Real Code-Switched ASR: A Spontaneous Nepali-English Podcast Benchmark

The paper introduces a new benchmark of 13,044 spontaneous Nepali‑English code‑switched podcast utterances, totaling 15.45 h, and evaluates a two‑stage synthetic‑to‑real adaptation framework for ASR. Stage 1 uses supervised fine‑tuning on 0.95 h of synthetic speech, reducing Mixed Error Rate from 23.40 % to 21.52 % and switch‑point error rate from 35.89 % to 28.77 %. Stage 2 applies Group Relative Policy Optimization with script and transcription rewards, yielding modest but statistically significant further improvements, particularly in English script confusion and switch‑point accuracy. The study also shows that an auxiliary script reward term offers no benefit beyond MER‑only optimization in this setting, and that the synthetic SFT provides the main cross‑domain transfer to spontaneous human speech.

By Firoj Paudel
nlpfine-tuningmultimodalbenchmarks
More like this →