arXiv:2610.16155v1 Announce Type: new
Abstract: Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlookin...
By Yutao Xie, Jingbo Yang, Guanyu Yao, Xiaoyu Yang, Duo Ding, Shiyu Chang
The paper introduces a three-tier persona vector for user simulation in evaluating LLM agents, comprising 23 dimensions across demographics, behavioral traits, and emotional states, plus a query-complexity overlay. It demonstrates that these nuanced personas generate diverse, scenario-reactive conversations, leading to significant variations in agent goal achievement and compliance across different contexts. The model’s design allows for reproducible, auditable user behavior patterns without relying on learned covariance matrices.
By Rahul Khedar, Eshita, Sneha Teja Sree Reddy Thondapu, Mayank Malhotra, Arup Kumar Das, Jitesh Chandra Mishra, Arun Menon, Avinash Karn, Mouli V
The paper introduces A-B-D, a method that infers personality traits of large language model agents from their behavioral data, combining functional actions and linguistic outputs across 345,667 real-world trajectories of 80 models. After filtering for stability and discriminability, 79 features are reduced to six reliable factors—two functional and four linguistic—revealing differences such as Claude‑Sonnet‑4.6’s planfulness versus GPT‑5.5’s low energy. The study also demonstrates a weak correlation between these inferred traits and agents’ self-reported Big Five scores, highlighting a knowledge‑action gap.
By Haokai Zhao, Jie Gao, Yunze Xiao, Xintao Wang, Weihao Xuan, Aditya Joshi, Mark Dredze, Jen-tse Huang
arXiv:2604.00477v2 Announce Type: replace
Abstract: LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessme...
By HyunJoon Jung, William Na
The paper introduces a three-tier persona vector to generate diverse, realistic user inputs for evaluating tool-augmented LLM agents. The vector includes 23 dimensions: categorical demographics, continuous behavioral traits, and continuous emotional states, plus a query-complexity overlay. Experiments on 64,698 conversations show that these persona dimensions produce measurable differences in agent performance and realistic scenario-reactive behavior.
arXiv:2510. 04491v3 Announce Type: replace Abstract: Despite rapid progress in building conversational AI agents, robustness is still largely untested.
By Muyu He, Anand Kumar, Tsach Mackey, Meghana Rajeev, James Zou, Nazneen Rajani