PersonaChatBench is a framework that evaluates conversational assistants by pairing the same task with multiple simulated users, each guided by a private persona card detailing distinct needs, preferences, and interaction behaviors. The framework measures task competence, responsiveness to diverse user needs, and simulator fidelity across 20 language models in six domains, revealing that performance varies by persona and domain. It highlights that strong task performance does not guarantee satisfaction of individual needs and that models’ strengths differ across contexts.
arXiv:2605.12894v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...
arXiv:2608. 19549v1 Announce Type: new Abstract: This paper addresses the issue of the significant labor required to test interview dialogue systems.
The paper introduces a three-tier persona vector for user simulation in evaluating LLM agents, comprising 23 dimensions across demographics, behavioral traits, and emotional states, plus a query-complexity overlay. It demonstrates that these nuanced personas generate diverse, scenario-reactive conversations, leading to significant variations in agent goal achievement and compliance across different contexts. The model’s design allows for reproducible, auditable user behavior patterns without relying on learned covariance matrices.
The paper introduces a three-tier persona vector to generate diverse, realistic user inputs for evaluating tool-augmented LLM agents. The vector includes 23 dimensions: categorical demographics, continuous behavioral traits, and continuous emotional states, plus a query-complexity overlay. Experiments on 64,698 conversations show that these persona dimensions produce measurable differences in agent performance and realistic scenario-reactive behavior.
arXiv:2607. 27816v2 Announce Type: replace-cross Abstract: Role-playing agents (RPAs) have become one of the most important consumer applications of large language models.
PersonaCodeBench is a new benchmark that evaluates coding agents in multi‑turn, persona‑driven sessions, turning user characteristics into verifiable obligations. The benchmark’s persona pool covers roles, technical abilities, interaction preferences, and practical constraints, generating task‑grounded, evolving requirements for implementation, permissions, testing, explanation, and delivery. Evaluation uses a controller to manage user simulator responses, online requirement evaluation, and separate verifiers for native correctness and persona fulfillment, with a joint pass condition requiring adherence to budget and validity conditions.
arXiv:2606. 18263v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to simulate human populations via persona prompting, often under the assumptions that richer persona descriptions improve behavioral fidelity, similarly sized attribute combinations are equally simulatable, and persona definitions generalize across tasks.
The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.
UXBench is the first user‑centric benchmark that uses real user feedback to evaluate AI assistants. It comprises three tasks—UX Judge, UX Eval, and UX Recovery—built from 7,400 test instances drawn from over 70,000 interaction logs of a mainstream Chinese AI assistant, covering 8 scenarios and 83 domains. Experiments on 26 state‑of‑the‑art language models reveal how model capability affects user experience and highlight six key findings about user feedback prediction and engagement.
arXiv:2609.22607v1 Announce Type: new Abstract: We argue here that the current dominant practice in LLM human simulation: prompting instruction-tuned assistant language models to role-play personas,...
The paper introduces StressSim, an intent‑preserving adversarial user simulator designed to stress‑test multi‑turn agents by generating challenging yet valid user interactions. StressSim optimises high‑level adversarial plans for each task instance using a genetic prompt‑optimisation loop that leverages rollout traces, task‑completion signals, and validity feedback to reduce task success while keeping the user’s goal unchanged. Experiments on MultiWOZ, retail, and airline domains show that StressSim reveals significant robustness gaps in leading LLM agents—reducing tau‑bench pass^4 scores by up to 69%—while maintaining realistic conversation quality and goal preservation with a modest optimisation budget.
PersonaForge is a user‑simulation framework that generates realistic multi‑turn interactions between users and agentic systems, addressing the gap that most training data assumes single‑turn queries. It uses a four‑dimensional persona space, SOUL‑driven behavioral control calibrated to real‑user statistics, and Reverse Deep Construction from authentic seed queries to create a 6.3K‑record training set and a 138‑task benchmark called PersonaForge‑Bench across 20 professional domains. Experiments with Qwen3.5‑27B show that training with PersonaForge improves composite scores by 4.1%, especially in Task Completion (+6.0%) and Response Quality (+6.8%), while also reducing turns and tool calls, indicating more efficient interactions.