arXiv:2609.21227v1 Announce Type: cross
Abstract: Factual hallucination is commonly defined by incorrect factual outputs. We study a paraphrase-induced hallucination setting, where a model answers a...
By Wenhan Yu, Wenxin Wu, Hao Wang, Lei Sha
The paper introduces Semantic Confusion to assess how consistently large language models refuse similar prompts. It presents ParaGuard, a 10k‑prompt corpus of controlled paraphrase clusters, and proposes three token‑level metrics—Confusion Index, Confusion Rate, and Confusion Depth—to measure contradictory refusal decisions across meaning‑preserving paraphrases. Experiments show that global false rejection rates can mask local inconsistencies, revealing that refusal evaluation must consider both frequency and consistency across nearby paraphrases.
By Riad Ahmed Anonto, Md Labid Al Nahiyan, Md Tanvir Hassan
arXiv:2606.16011v2 Announce Type: replace
Abstract: Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain...
By Nafiseh Nikeghbal, Amir Hossein Kargaran, Shaghayegh Kolli, Jana Diesner
PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.
By Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov, Valeriia Kuschenko, Alexander Evseev, Sergey Bolovtsov
arXiv:2608. 10315v1 Announce Type: cross Abstract: Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching.
By Siyang Wu, Yibo Jiang, Bryon Aragam
The paper investigates token‑level certainty as a proxy for correctness in large language models. It finds that certainty better predicts whether a model will answer a question correctly than it does whether a specific response is correct, and that certainty varies by token type and position. The authors show that using certainty early in generation to allocate responses and later to weight votes improves accuracy while dramatically cutting token cost.
By Yunfan Zhou, Ye Zhu, Zhihai Wang, Jianguo Yao, Haibing Guan, Xijun Li