The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination

The paper presents a rate‑distortion framework for understanding factual hallucination in closed‑book question answering. It shows that even when a fact is observed, limited memory forces it to be stored approximately, leading to errors that can be bounded by a combination of compression distortion and missing coverage. The authors derive a theoretical lower bound on error and validate it with simulations and probes on modern language models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• nlp
• safety