arXiv AI
"Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulators
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 19
KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn
KnowSim introduces an evaluation framework that uses a user simulator with explicit knowledge states to assess how well large language models calibrate information to users. The simulator represents knowledge as a graph of Information Units with prerequisite relationships and updates these states based on learning theory. KnowSim computes Knowledge Gain, Delivery Calibration, and Cognitive Overload metrics, and its rankings align with human judgments, outperforming baseline simulators and revealing model performance differences across user knowledge levels.
arXiv AI
Jun 16
Edu-Theater: A Data-Efficient Agent Framework for Scalable Learner Behavior Simulation through Staging Roll-Call
arXiv:2606. 15225v1 Announce Type: cross Abstract: Large-scale learner-task interaction data are crucial for intelligent educational systems but are costly to collect and constrained by privacy and learner engagement.
arXiv AI
Jul 24
AI Assistants Overassist
arXiv:2607. 21306v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as tutors and thought partners, helping users reason through problems.
arXiv Machine Learning
Aug 4
MemoryForge: Synthesize Lifelong Memory for Human-Like LLM Agents
arXiv:2608. 00007v1 Announce Type: cross Abstract: Equipping Large Language Models (LLMs) with human-like personas is crucial for agentic applications, such as role-play and user simulation.
arXiv AI
Aug 12
INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators
arXiv:2608. 10492v1 Announce Type: new Abstract: Large Language Model (LLM)-based simulators often reproduce observable actions but fail to capture the underlying reasoning behind them.
arXiv Computation and Language
2d ago
Investigating Assistant Bias in LLM User Simulators Using a Role Vector
arXiv:2609.00608v1 Announce Type: new Abstract: LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, the...