arXiv:2606. 10029v1 Announce Type: cross Abstract: Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream.
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale.
arXiv:2606. 07473v1 Announce Type: cross Abstract: Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input.
The paper introduces an Encoding Probe that reconstructs language model representations using interpretable features, addressing limitations of traditional decoding probes such as incomparable feature contributions and correlation effects. It evaluates this approach on text and speech transformer models, examining features from acoustics, phonetics, syntax, lexicon, and speaker identity. Findings reveal that speaker-related effects vary with training objectives and datasets, while syntactic and lexical features independently contribute to reconstruction, offering a complementary perspective on model interpretation.
arXiv:2606. 14647v1 Announce Type: cross Abstract: Transformer-based automatic speech recognition (ASR) models such as Whisper are highly accurate, but their predictions remain difficult to interpret.
arXiv:2509.10452v3 Announce Type: replace-cross Abstract: Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance....
arXiv:2609.07876v1 Announce Type: cross Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...
The paper investigates how encoder-decoder models in ASR and NMT can generate fluent text even when the input contains no recoverable message, a phenomenon known as message-free hallucination. By auditing the models’ reserved null tokens and manipulating their scores, the authors show that a higher null-token score can suppress fabrication but may also delete valid content or shorten translations. The study highlights that the null token can serve as a diagnostic tool for hallucination and suggests evaluating abstention methods by considering both suppression and deletion costs.
arXiv:2609.36913v1 Announce Type: cross Abstract: Transcribing domain-specific entities and rare proper nouns remains a major challenge in automatic speech recognition (ASR). In this paper, we propos...
arXiv:2609.13151v1 Announce Type: new Abstract: Leading multilingual speech recognition models like Whisper transcribe diverse, low-resource languages without language-specific training but are compu...
The paper introduces a new strategy for connecting large language models (LLMs) to speech encoders in automatic speech recognition (ASR) systems by sharing a single connector across languages within the same linguistic family. This approach reduces the number of parameters needed compared to training a separate connector for each language, while improving generalization across different domains and real‑world corpora. Experiments with two multilingual LLMs and two speech datasets demonstrate that family‑based connectors are both efficient and effective for multilingual ASR deployment.
The paper examines how multilingual medical adaptation affects the internal representations of Whisper ASR models. By comparing various fine‑tuning strategies—zero‑shot decoding, English‑only, German‑only, two‑stage EN→EN+DE, and direct EN+DE fine‑tuning—it shows that fine‑tuning significantly improves performance, with the best model varying by setting. Layer‑wise encoder analysis reveals that English medical fine‑tuning drives the main representation shift, while multilingual continuation largely preserves the adapted space, and that domain and language signals remain recoverable across layers.