arXiv Computation and Language October 9, 2026 By Mengze Hong, Zeyang Lei, Wenbo Shang, Xia Zeng, Xiying Zhao, Qi Zhu, Chen Jason Zhang, Di Jiang, Taiming Fu, Qiongyi Zhou, Qinghe Chang, Fubao Zhang, Chenxuan Ma, Minlong Peng, Jinfeng Huang, Zineng Zhou, Jindou Wu, Muge Qi, Sijun He, Xin Cui, Di Liang, Yuan Hua, Davey Chen

UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions

Read the original on arXiv Computation and Language →

UXBench Pro is a new benchmark that evaluates personalized user experience in multi-turn dialogue interactions. It contains 1,000 test instances from real user interactions across 12 task scenarios and 82 domains, each paired with a FACTORS user profile describing seven behavioral facets. The benchmark introduces a dual-perspective evaluation: a personalized User Reward Model for third‑person judgment and Sim4Eval, a user simulator for first‑person evaluation across four cognitive state dimensions, along with meta‑benchmarks URMBench and USimBench to assess evaluator reliability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Aug 3

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

arXiv:2607. 27816v2 Announce Type: replace-cross Abstract: Role-playing agents (RPAs) have become one of the most important consumer applications of large language models.

By Yuhang Zhu, Mingxuan Du, Benfeng Xu, Jie Gao, Lingyun Yu, Hongtao Xie

llms agents benchmarks

arXiv Machine Learning

Jul 31

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604. 07343v2 Announce Type: replace-cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.

By Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 30

Synthetic Interaction Data for Scalable Personalization in Large Language Models

arXiv:2602. 12394v2 Announce Type: replace Abstract: Personalized prompting offers large opportunities for deploying large language models (LLMs) to diverse users, yet existing prompt optimization methods primarily focus on task-level optimization while largely overlooking user-specific preferences and latent constraints of individual users.

By Yuchen Ma, Yue Huang, Wenjie Wang, Xiaonan Luo, Xiangliang Zhang, Stefan Feuerriegel

llms agents reinforcement-learning benchmarks safety

arXiv Computation and Language

Oct 12

Training User Simulators with Latent Intrinsic Feedback

The paper introduces user simulators that provide intrinsic feedback by incorporating latent evaluation states such as internal thoughts, reactions, and scalar ratings. These simulators, trained on real user–assistant interaction logs, outperform prompted baselines in realism and critiquing behaviors, and their turn‑level evaluations better align with human judgments. Additionally, the intrinsic feedback serves as an effective reward signal for multi‑turn reinforcement learning, reducing compute requirements by over 50% compared to setups using external judge LLMs.

By Tarek Naous, Craig Boutilier, Wei Xu, Ofer Meshi

llms reinforcement-learning fine-tuning

arXiv AI

Sep 1

Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions

arXiv:2603.04191v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as personal assistants, where users may share individual preferences over extended interactio...

By Qianyun Guo, Yibo Li, Yue Liu, Bryan Hooi

llms benchmarks

arXiv Computation and Language

Oct 5

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.

By Anjali Kantharuban, Jonas Mueller

llms rag agents benchmarks safety