KnowSim introduces an evaluation framework that uses a user simulator with explicit knowledge states to assess how well large language models calibrate information to users. The simulator represents knowledge as a graph of Information Units with prerequisite relationships and updates these states based on learning theory. KnowSim computes Knowledge Gain, Delivery Calibration, and Cognitive Overload metrics, and its rankings align with human judgments, outperforming baseline simulators and revealing model performance differences across user knowledge levels.
By Yoonjoo Lee, Hyoungwook Jin, Tae Soo Kim, Shaoyang Zhang, Philippe Laban, Q. Vera Liao
arXiv:2606. 15225v1 Announce Type: cross Abstract: Large-scale learner-task interaction data are crucial for intelligent educational systems but are costly to collect and constrained by privacy and learner engagement.
By Weibo Gao, Qi Liu, Linan Yue, Zheng Zhang, Yichao Du, Fangzhou Yao, Ao Yu, Zhenya Huang, Shijin Wang
arXiv:2607. 21306v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as tutors and thought partners, helping users reason through problems.
By Verona Teo, Raghav Jain, Tobias Gerstenberg, Max Kleiman-Weiner
arXiv:2608. 00007v1 Announce Type: cross Abstract: Equipping Large Language Models (LLMs) with human-like personas is crucial for agentic applications, such as role-play and user simulation.
By Bohan Tang, Yiwen Guo
arXiv:2608. 10492v1 Announce Type: new Abstract: Large Language Model (LLM)-based simulators often reproduce observable actions but fail to capture the underlying reasoning behind them.
By Rose Niousha, Minwoo Kang, Narges Norouzi
The paper introduces SWIM, a task that frames student writing simulation as proficiency‑conditioned essay generation. It evaluates prompting, supervised fine‑tuning, and reinforcement learning for aligning generated essays with student proficiency profiles, using automated essay scoring as a metric. Results show that prompting alone offers limited control, while supervised fine‑tuning and reinforcement learning significantly improve alignment across content, lexical, grammatical, and organizational traits, though low‑proficiency writing remains difficult to replicate.
By Heejin Do, Jakub Kontak, Mrinmaya Sachan
arXiv:2609.00608v1 Announce Type: new
Abstract: LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, the...
By Daeheon Jeong, Yoonjoo Lee, Eugene Choi, Sinie van der Ben, Juho Kim
OmniSimulator is a training framework that enhances small language models for realistic user simulation on short‑video platforms. It structures behavior‑oriented reasoning into five dimensions, selects real user‑behavior data, and uses a teacher model to generate high‑quality reasoning traces, enabling small models to learn behavior‑relevant patterns instead of superficial correlations. The approach yields a 34.33% performance boost for Qwen3‑8B, competitive with Claude‑Sonnet‑4.5, and supports smaller‑scale, self‑hosted deployment.
By Jiawei Chen, Ruoxi Xu, Boxi Cao, Zhuoqun Li, Ruotong Pan, Yunfei Zhang, Zhirui Yang, Jingyan Chen, Tingting Gao, Han Li, Yaojie Lu, Xianpei Han, Le Sun, Xiangyu Wu, Hongyu Lin
The paper evaluates Large Language Models for automatically analyzing responses in digital teacher simulations. Experiments compare DeBERTaV3 and Llama 3 across zero‑shot, few‑shot, and fine‑tuning settings, revealing that performance varies by characteristic and that Llama 3 consistently outperforms DeBERTaV3, especially when new characteristics must be identified. The findings suggest Llama 3 is preferable for dynamic simulation environments where teacher educators introduce new evaluation criteria.
By David de-Fitero-Dominguez, Mariano Albaladejo-Gonz\'alez, Antonio Garcia-Cabot, Eva Garcia-Lopez, Antonio Moreno-Cediel, Erin Barno, Justin Reich
The paper introduces ALPS, a hierarchical simulated student model that alternates between planning profile‑aligned task segments and generating corresponding actions, aiming to maintain consistent student behavior over long‑horizon learning environments. Unlike existing LLM‑based simulators that focus on short, language‑centered interactions, ALPS is conditioned on observable, environment‑specific behavior profiles and evaluated in both data‑sufficient and data‑scarce settings. Experiments in an inquiry‑based physics game and a pharmacy simulation show that ALPS consistently outperforms turn‑level generation, achieving comparable behavioral density and judge pass rates even with limited real student data, though coverage remains lower.
By Bahar Radmehr, Jiaxu Zhao, Fatma Bet\"ul G\"ure\c{s}, Adish Singla, Tanja K\"aser
ARGUS is a neuro‑symbolic, agenda‑driven framework designed to improve the reliability of user simulators in interactive environments. It separates state tracking from natural language generation by using a dynamic agenda stack and symbolic operations to enforce goal progression and constraint‑aligned interactions. The authors distill the model into compact versions via evaluator‑guided SFT and online RL, and evaluate its behavioral fidelity with a new Composite Behavioral Score metric on $ au^3$‑Bench and MultiWOZ, showing significant performance gains.
By Hang Zheng, Nan Jiang, Zichen Zhu, Zixu Yang, Zhiyang Tang, Shuai Fan, Lu Chen, Xiaobao Wu, Kai Yu
arXiv:2605.12894v2 Announce Type: replace-cross
Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...
By Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques