The paper introduces a training‑free, inference‑time technique to curb hallucinated transcripts in Whisper by projecting decoder activations away from a low‑rank hallucination subspace derived from non‑speech data. Two variants are tested: an always‑on projection that dramatically lowers hallucination rates on non‑speech benchmarks, and a gated version that applies the projection only when non‑speech is predicted, achieving a smaller but still significant reduction. On LibriSpeech, the gated method slightly increases word error rate but keeps false‑rejection rates low, demonstrating a controllable trade‑off between hallucination suppression and recognition accuracy.
arXiv:2608. 00722v1 Announce Type: cross Abstract: Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text.
arXiv:2606. 07473v1 Announce Type: cross Abstract: Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input.
arXiv:2511. 13300v1 Announce Type: cross Abstract: Generative models have shown remarkable performance in speech enhancement (SE), achieving superior perceptual quality over traditional discriminative approaches.
The paper investigates why automatic speech recognition (ASR) systems sometimes generate fluent text that does not correspond to the input audio, a phenomenon termed hallucination. By examining two independently trained Conformer‑Large models—one using CTC and the other RNN‑T—under conditions of environmental noise and speaker‑background shift, the authors identify the final encoder stage as a critical boundary. Bypassing this final block leads to divergence on almost all utterances, while bypassing earlier blocks has minimal effect; at this stage, representations become compact, the decoder can read the text, and grapheme information becomes explicit, yet the output is garbled or repetitive rather than fluent fabrication. The study thus pinpoints a mechanistic precondition for hallucination—failure to produce adequately grounded output—though it does not fully explain naturally occurring hallucinations, and it highlights a consistent terminal‑stage dependency across decoder families and distribution shifts.
The paper investigates how encoder-decoder models in ASR and NMT can generate fluent text even when the input contains no recoverable message, a phenomenon known as message-free hallucination. By auditing the models’ reserved null tokens and manipulating their scores, the authors show that a higher null-token score can suppress fabrication but may also delete valid content or shorten translations. The study highlights that the null token can serve as a diagnostic tool for hallucination and suggests evaluating abstention methods by considering both suppression and deletion costs.
The paper investigates whether the internal hidden states of large language models can improve the detection of hallucinations in model responses. By integrating hidden states with an output‑side baseline and evaluating on both open‑ended generation and multiple‑choice tasks, the authors find that adding hidden states yields modest AUROC gains of about 0.004 and 0.039. These results suggest that while hidden states contain additional predictive information, the overall improvement over strong standalone detection methods remains limited.
arXiv:2606. 02642v1 Announce Type: cross Abstract: Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination.
The paper introduces a trilingual spoken hallucination detection benchmark covering English, Russian, and Kazakh news, with 12,013 samples that include synthetic alterations and severity levels, as well as 290 fact‑checked misinformation items. Detectors are evaluated in a reference‑free setting on text, ASR transcripts, and audio, revealing that most models underperform baseline classifiers, except Gemma‑3n on transcripts. Synthetic‑trained detectors achieve high macro‑F1 scores on real‑world misinformation, but Russian provenance analysis highlights model‑dependent signals that confound synthetic benchmarks.
arXiv:2504. 10020v4 Announce Type: replace-cross Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs).
arXiv:2510.12851v2 Announce Type: replace-cross Abstract: Large Audio-Language Models (LALMs) excel in Audio QA but often suffer from hallucinations ungrounded in the audio. To our knowledge, we are...
The paper revisits large‑language‑model (LLM) based automatic speech recognition (ASR) from an entropy‑allocation perspective, introducing three metrics to analyze how entropy reduction is distributed between the speech encoder and the LLM. It proposes a multi‑stage training strategy that addresses entropy‑allocation inefficiencies, including a redesigned pretraining phase to reduce the speech‑text modality gap and an iterative asynchronous fine‑tuning stage to maintain functional decoupling and limit encoder drift. Experiments on Mandarin and English benchmarks demonstrate competitive performance with state‑of‑the‑art models using only 2.3 B parameters, while also reducing hallucinations through the decoupling‑oriented design.
arXiv:2606. 00819v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved strong performance across diverse natural language tasks, yet their outputs often suffer from hallucinations -- content that is misaligned with factual information.