arXiv AI By Griffin Farrow, Lily Sijia Li, Jack Johnson, Tingyan Wang, Philip Torr, William Bolton, Fabio J. Fehr

When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation

Read the original on arXiv AI →

The paper investigates how rubric-based evaluations of medical language models fail to detect hallucinations that could undermine clinician trust. Using a controlled study with MedHallu and a clinician-validated error-injection pipeline, the authors show that more specific rubrics can distinguish correct from hallucinated responses, yet many clinically relevant hallucinations remain undetected across HealthBench, HealthBench Professional, and LiveMedBench. The study finds that rubrics work best when explicitly checking facts but miss additional or unexpected errors, and suggests that a retrieval-based factuality check can recover some of these blind spots.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 28

Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention

The paper surveys hallucinations in large language models (LLMs) through a lifecycle lens, covering causes, detection, mitigation, and prevention. It categorizes hallucinations into data‑related, training‑related, and inference‑related stages, aligning each with specific interventions. The authors also review benchmark datasets and propose a standardized framework to diagnose and address hallucinations for safer, more reliable LLMs.

By Naveen Lamba, Sanju Tiwari, Manas Gaur
arXiv AI
Aug 5

Quantifying Hallucinations in Language Language Models on Medical Textbooks

arXiv:2603. 09986v3 Announce Type: replace-cross Abstract: Hallucinations, the tendency for large language models to provide responses with factually incorrect and unsupported claims, is a serious problem within natural language processing for which we do not yet have an effective solution to mitigate against.

By Brandon C. Colelough, Davis Bartels, Dina Demner-Fushman
arXiv Machine Learning
Sep 14

When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs

The paper examines a specific type of hallucination in large language models caused by spurious correlations—unintended, statistically prominent associations in training data such as surnames linked to nationalities. These hallucinations are confidently produced, persist regardless of model scaling or refusal fine‑tuning, and evade existing detection methods like confidence filtering and inner‑state probing. The authors use controlled synthetic experiments and evaluations on both open‑source and proprietary LLMs, including GPT‑5, to demonstrate the failure of current detection techniques and provide a theoretical explanation for why statistical biases undermine confidence‑based approaches.

By Shaowen Wang, Yiqi Dong, Ruinian Chang, Tansheng Zhu, Yuebo Sun, Kaifeng Lyu, Jian Li