The paper demonstrates that open‑ended Theory‑of‑Mind trackers can produce valid beliefs that are absent from finite reference sets, and that treating unmatched outputs as false can reverse model‑selection rankings. By recoding references for 259 beliefs, the authors show a dramatic drop in weighted prevalence and a reversal of strictly proper Brier risk, with similar distortions observed in a 301‑question NQ‑open DPR‑BERT pipeline. The study further reveals that 90‑96% of audited unmatched beliefs are literally true, and introduces a TriSource‑Restore method that anchors reference labels to a probability‑sampled human pilot to restore calibration and ranking integrity.
By Zhexi Feng, Wuxi Chen, Bingrui Zhang
The paper audits a 366‑day autobiographical book generated by a large language model (LLM) against an independent verification corpus. Using a four‑level rubric, 354 of the 366 days (96.7%) failed verification, with only 12 days containing corroborated scenes and 19 days containing actively contradicted claims. Regenerating the same days with current models yielded 100% verification failure, while grounding the generation in the subject’s own corpus improved the rate to 83.3% but still left substantial residual failure.
By Heather Renze
The paper revises earlier claims about hallucination rates in multi‑agent LLM systems, noting that a judge that flags contradictions can mistakenly label correct reports of teammate errors as hallucinations. Using a pre‑registered analysis plan, the authors re‑score outputs from a harness with Llama and Claude agents, finding that the broadcast effect on hallucination rates is negligible and that most hallucinations stem from teammates’ wrong values. The study also reports that the effect varies across judge‑prompt combinations and that the data are limited to a single harness and 13 hand‑written scenarios.
By Carson Rodrigues
arXiv:2609.25447v1 Announce Type: new
Abstract: We study what LLMs do when a user applies pressure in an uncomfortable situation: a user insists, begs, flatters or grieves, and the model gives up a c...
By Tapan Parikh
arXiv:2610.20417v1 Announce Type: new
Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
By Jiahong Qian
arXiv:2606. 10315v1 Announce Type: cross Abstract: LLM-as-judge is the default instrument for evaluating conversational agents, yet its reliability is almost always reported as agreement with human ratings, not recall of real defects.
By Sawyer Zhang, Alexander Wang, Sophie Lei