arXiv Computation and Language
Investigating Assistant Bias in LLM User Simulators Using a Role Vector
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Oct 12
Training User Simulators with Latent Intrinsic Feedback
The paper introduces user simulators that provide intrinsic feedback by incorporating latent evaluation states such as internal thoughts, reactions, and scalar ratings. These simulators, trained on real user–assistant interaction logs, outperform prompted baselines in realism and critiquing behaviors, and their turn‑level evaluations better align with human judgments. Additionally, the intrinsic feedback serves as an effective reward signal for multi‑turn reinforcement learning, reducing compute requirements by over 50% compared to setups using external judge LLMs.
arXiv Computation and Language
Oct 8
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
arXiv:2605.12894v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...
arXiv AI
Jun 15
OdysSim: Building Foundation Models for Human Behavior Simulation
arXiv:2606. 14199v1 Announce Type: cross Abstract: Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.
arXiv AI
Sep 10
A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation
The paper introduces a three-tier persona vector for user simulation in evaluating LLM agents, comprising 23 dimensions across demographics, behavioral traits, and emotional states, plus a query-complexity overlay. It demonstrates that these nuanced personas generate diverse, scenario-reactive conversations, leading to significant variations in agent goal achievement and compliance across different contexts. The model’s design allows for reproducible, auditable user behavior patterns without relying on learned covariance matrices.
arXiv Computation and Language
Oct 5
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.
Hugging Face Trending Papers
Sep 8
A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation
The paper introduces a three-tier persona vector to generate diverse, realistic user inputs for evaluating tool-augmented LLM agents. The vector includes 23 dimensions: categorical demographics, continuous behavioral traits, and continuous emotional states, plus a query-complexity overlay. Experiments on 64,698 conversations show that these persona dimensions produce measurable differences in agent performance and realistic scenario-reactive behavior.