KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn

KnowSim introduces an evaluation framework that uses a user simulator with explicit knowledge states to assess how well large language models calibrate information to users. The simulator represents knowledge as a graph of Information Units with prerequisite relationships and updates these states based on learning theory. KnowSim computes Knowledge Gain, Delivery Calibration, and Cognitive Overload metrics, and its rankings align with human judgments, outperforming baseline simulators and revealing model performance differences across user knowledge levels.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms