← Back to all news
arXiv AI By Maryam Abbasihafshejani, Murtuza Jadliwala

Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

Read the original on arXiv AI →

The paper introduces a training‑free, inference‑time technique to curb hallucinated transcripts in Whisper by projecting decoder activations away from a low‑rank hallucination subspace derived from non‑speech data. Two variants are tested: an always‑on projection that dramatically lowers hallucination rates on non‑speech benchmarks, and a gated version that applies the projection only when non‑speech is predicted, achieving a smaller but still significant reduction. On LibriSpeech, the gated method slightly increases word error rate but keeps false‑rejection rates low, demonstrating a controllable trade‑off between hallucination suppression and recognition accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
6d ago

The Anatomy of an ASR Hallucination

The paper investigates why automatic speech recognition (ASR) systems sometimes generate fluent text that does not correspond to the input audio, a phenomenon termed hallucination. By examining two independently trained Conformer‑Large models—one using CTC and the other RNN‑T—under conditions of environmental noise and speaker‑background shift, the authors identify the final encoder stage as a critical boundary. Bypassing this final block leads to divergence on almost all utterances, while bypassing earlier blocks has minimal effect; at this stage, representations become compact, the decoder can read the text, and grapheme information becomes explicit, yet the output is garbled or repetitive rather than fluent fabrication. The study thus pinpoints a mechanistic precondition for hallucination—failure to produce adequately grounded output—though it does not fully explain naturally occurring hallucinations, and it highlights a consistent terminal‑stage dependency across decoder families and distribution shifts.

By Hamees Sayed, Apoorv Singh, Kumar Aman, Akshat Mandloi
multimodalsafety
More like this →
arXiv Machine Learning
Aug 18

The Null Token Knows: Reducing Message-Free Hallucination in ASR and NMT

The paper investigates how encoder-decoder models in ASR and NMT can generate fluent text even when the input contains no recoverable message, a phenomenon known as message-free hallucination. By auditing the models’ reserved null tokens and manipulating their scores, the authors show that a higher null-token score can suppress fabrication but may also delete valid content or shorten translations. The study highlights that the null token can serve as a diagnostic tool for hallucination and suggests evaluating abstention methods by considering both suppression and deletion costs.

By Kirill Borodin, Vasiliy Kudryavtsev, Ivan Viakhirev
multimodalsafety
More like this →