The paper introduces a drill‑down evaluation framework that breaks down Text‑to‑SQL queries into simpler executable sub‑queries and natural‑language questions, enabling analysis of hallucination patterns along the generation path. It distinguishes recurrent hallucinations—those appearing in both early and final steps—from emergent hallucinations that only surface at the end. Experiments on five large language models and an expanded BIRD‑mini benchmark reveal varying recurrence and emergence probabilities across categories, with conversational history generally increasing recurrence and decreasing emergence.
arXiv:2601. 02023v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora.
arXiv:2609.21509v1 Announce Type: new Abstract: When language models reason in chain-of-thought or exchange free-text intermediates, they serialize structured information into natural language. How m...
arXiv:2608. 07525v1 Announce Type: cross Abstract: Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applications.
DialFork is a new benchmark that evaluates procedural hallucination in conversational agents, a failure mode where agents produce fluent, factually plausible utterances but deviate from the intended dialogue process. The benchmark consists of 36,278 dialogues across 24 tasks in 12 domains, created by forking conformant dialogues and injecting one of four hallucination types, with validation by 30 human annotators. Three evaluation tasks are defined—monitoring overall deviation, procedural retrieval of a single dialogue, and real‑time detection of off‑track turns—using dialog‑flow‑aware metrics FlowKL and FlowScore, which outperform baseline semantic similarity metrics.
arXiv:2606. 07537v1 Announce Type: cross Abstract: Large language models hallucinate--producing fluent, confident, factually wrong outputs--with a consistency that persists across generations and scales.
arXiv:2607. 04223v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why.
The paper explores using a language model’s low‑level symbolic skill—specifically SQL—to detect hallucinations without fine‑tuning. By having the model construct an SQL database from reference documents, it can reason over both the source and the model’s output, creating a neurosymbolic check. Experiments on RAGTruth and DiaHalu show this method outperforms direct prediction and rivals state‑of‑the‑art detectors, highlighting the value of leveraging inherent symbolic competences in LLMs.
In 2026, the SHROOM-Visions shared task was launched at the UncertaiNLP Workshop co‑located with EMNLP to address hallucinations in large vision‑language models. The task builds on the SHEEP dataset and asks participants to detect and classify fine‑grained hallucination spans in image‑conditioned text generation across four languages (Chinese, English, French, Italian) using a five‑class taxonomy. The competition attracted 27 teams and over 600 system submissions, with top systems achieving character‑level, label‑conditioned, and IoU scores of 0.58, 0.46, and 0.51 respectively, surpassing baselines by 30‑40 points.
arXiv:2607. 20489v1 Announce Type: new Abstract: Text-to-SQL has advanced rapidly with large language models, but complex database queries still require reasoning beyond one-shot generation, including multi-step decomposition, execution-based diagnosis, and targeted correction.
The paper introduces SAFE‑QE, a training‑free, retriever‑agnostic method for query expansion that mitigates hallucinated terms from large language models. It uses a subsequence‑association filter to score candidate terms based on their reproducibility across randomised query contexts, down‑weighting terms that lack corpus support. SAFE‑QE then selects a broad yet consistent set of expansions via constrained coverage maximisation, and defines a new metric, the Hallucinated Expansion Term Rate, to quantify faithfulness. The approach matches state‑of‑the‑art ranking performance while injecting far fewer unsupported terms on TREC and BEIR benchmarks.
arXiv:2606. 17821v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in translating natural language to SQL, yet existing methods still falter on complex queries requiring multi-step, data-aware reasoning.
In 2026, we held the fourth iteration of the SHROOM Shared Task series: SHROOM-Visions (\textbf{S}hared-task on \textbf{H}allucinations and \textbf{R}elated \textbf{O}bservable \textbf{O}vergeneration...