arXiv AI

The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models

arXiv AI
Aug 5

Quantifying Hallucinations in Language Language Models on Medical Textbooks

arXiv:2603. 09986v3 Announce Type: replace-cross Abstract: Hallucinations, the tendency for large language models to provide responses with factually incorrect and unsupported claims, is a serious problem within natural language processing for which we do not yet have an effective solution to mitigate against.

By Brandon C. Colelough, Davis Bartels, Dina Demner-Fushman
arXiv Computation and Language
Sep 2

Reliability Challenges in Diffusion Vision-Language Models

The paper presents the first systematic reliability evaluation of diffusion-based Large Vision‑Language Models (dLVLMs), comparing six diffusion models to autoregressive (AR) baselines across four dimensions. Key findings include a reversal of the yes‑bias seen in AR models for binary visual queries, competitive hallucination rates but lower linguistic quality, near‑zero accuracy for underrepresented racial groups with opposite‑polarity gender bias, and accuracy collapse in multiple‑choice tasks when the correct option is shorter than distractors due to a length prior emerging at the first denoising step. Additionally, tokens committed late in denoising with low confidence correlate with hallucinated content, indicating a unique mechanistic signal in diffusion generation.

By Md. Atabuzzaman, Chris Thomas
arXiv Computation and Language
Oct 9

Fact over Fiction: Detection of Pathological Hallucinations in Sinhala-to-English Neural Machine Translation

The paper presents a reference‑free framework for detecting hallucinations in Sinhala‑to‑English neural machine translation. It introduces a 45,000‑sample synthetic dataset created via five linguistically motivated corruption strategies and a semantic rescue mechanism that uses character‑level similarity. Using a fine‑tuned mDeBERTa‑v3 model, the authors achieve a token‑level F1 of 0.841 and demonstrate that the detector relies on the Sinhala source, not on surface artefacts, with an AUROC of 0.970 on a source‑disjoint test set. The study also benchmarks eight NMT systems, showing that hallucination rates vary by an order of magnitude across architectures.

By Navam Obeysekara, Nevidu Jayatilleke