arXiv AI By Julia Belikova, Elisei Rykov

Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

Read the original on arXiv AI →

The paper presents a meta‑study of hallucination detection in large language models, framing it as a benchmark‑conditioned measurement problem. It organizes existing benchmarks along three axes—provenance, typology, and granularity—to analyze how construction, annotation, and evaluation choices influence detection tasks. The authors also examine span‑level evaluation, propose a new Span Coverage F1 metric, and discuss the implications for cross‑benchmark comparisons and generalization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
6d ago

SHAM-Bench: A Span-level Hallucination Benchmark Across Multiple Tasks

The paper introduces SHAM-Bench, a unified benchmark for detecting span-level hallucinations across six natural language generation tasks, compiled from 6,461 human-annotated examples under a common schema. It evaluates existing detection methods, finding that fine‑tuned detectors perform well in‑domain but falter under distribution shift, while prompt‑based methods with reasoning show greater robustness. The study also highlights challenges such as subtle unsupported content, ambiguous span boundaries, and noisy labels, and releases the dataset, evaluation code, and a public leaderboard for future research.

By Sicong Huang, Ian Lane
arXiv Computation and Language
6d ago

Detecting Errors in Large Language Model Responses: The Incremental Predictive Value of Hidden States

The paper investigates whether the internal hidden states of large language models can improve the detection of hallucinations in model responses. By integrating hidden states with an output‑side baseline and evaluating on both open‑ended generation and multiple‑choice tasks, the authors find that adding hidden states yields modest AUROC gains of about 0.004 and 0.039. These results suggest that while hidden states contain additional predictive information, the overall improvement over strong standalone detection methods remains limited.

By Ningzhi Li, Xiaojun Ma, Xiaofei Wang