Large Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios.
UXBench Pro is a new benchmark that evaluates personalized user experience in multi-turn dialogue interactions. It contains 1,000 test instances from real user interactions across 12 task scenarios and 82 domains, each paired with a FACTORS user profile describing seven behavioral facets. The benchmark introduces a dual-perspective evaluation: a personalized User Reward Model for third‑person judgment and Sim4Eval, a user simulator for first‑person evaluation across four cognitive state dimensions, along with meta‑benchmarks URMBench and USimBench to assess evaluator reliability.
By Mengze Hong, Zeyang Lei, Wenbo Shang, Xia Zeng, Xiying Zhao, Qi Zhu, Chen Jason Zhang, Di Jiang, Taiming Fu, Qiongyi Zhou, Qinghe Chang, Fubao Zhang, Chenxuan Ma, Minlong Peng, Jinfeng Huang, Zineng Zhou, Jindou Wu, Muge Qi, Sijun He, Xin Cui, Di Liang, Yuan Hua, Davey Chen
arXiv:2608.02372v2 Announce Type: replace
Abstract: Large language model (LLM) agents are increasingly used to support decisions by calling external tools. Existing work on LLM tool-calling assumes t...
By Abdulrahman AlRabah, Xiaocheng Yang, Dilek Hakkani-T\"ur, Abdussalam Alawini
arXiv:2607. 20485v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable performance on standard benchmarks, yet it remains largely unexplored whether they truly meet user expectations.
By Miaomiao Li, Yang Wang, Bin Liang, Shudong Liu, Zhiwei Zhang, Kam-Fai Wong
arXiv:2610.09163v1 Announce Type: new
Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when...
By Arkajyoti Chakraborty, Aryan Tayal, Ishika Agarwal, Tanner Sorensen, Justin Chiu, Alessandro Di Bari, Neha Gupta, Andreas Stolcke
arXiv:2607. 20558v1 Announce Type: cross Abstract: AI Assistants are increasingly deployed in high-stakes settings, such as healthcare or government services.
By Emma Kondrup, Zachary Yang, Anne Imouza, Reihaneh Rabbany