arXiv Computation and Language
Investigating Assistant Bias in LLM User Simulators Using a Role Vector
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jun 15
OdysSim: Building Foundation Models for Human Behavior Simulation
arXiv:2606. 14199v1 Announce Type: cross Abstract: Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.
arXiv AI
Aug 3
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
arXiv:2607. 27816v2 Announce Type: replace-cross Abstract: Role-playing agents (RPAs) have become one of the most important consumer applications of large language models.
arXiv AI
Jun 4
Unlocking Proactivity in Task-Oriented Dialogue
arXiv:2605. 22240v2 Announce Type: replace Abstract: Proactive task-oriented dialogue (TOD), such as outbound sales, demands a persuasive agent that actively probes the user's concerns and steers the conversation toward acceptance within a bounded number of turns.
arXiv AI
Aug 19
KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn
KnowSim introduces an evaluation framework that uses a user simulator with explicit knowledge states to assess how well large language models calibrate information to users. The simulator represents knowledge as a graph of Information Units with prerequisite relationships and updates these states based on learning theory. KnowSim computes Knowledge Gain, Delivery Calibration, and Cognitive Overload metrics, and its rankings align with human judgments, outperforming baseline simulators and revealing model performance differences across user knowledge levels.
arXiv Computation and Language
Aug 21
Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems
arXiv:2608. 19549v1 Announce Type: new Abstract: This paper addresses the issue of the significant labor required to test interview dialogue systems.
arXiv AI
Jun 6
PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation
arXiv:2606. 05697v1 Announce Type: new Abstract: User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly.