The paper introduces TextQ‑German, a dataset suite for evaluating German natural language generation (NLG) from a Quality of Experience (QoE) perspective, covering tasks such as summarization and machine translation. Human ratings collected via crowdsourcing identify perceptual quality dimensions, and the authors develop automatic QoE prediction models—including transformer‑based, linguistic feature‑based, and hybrid approaches—showing that hybrid models outperform pure transformers and that linguistic features alone can rival fine‑tuned language models. The dataset is further enriched with large language model (LLM) outputs annotated with overall QoE scores, and validation on held‑out data demonstrates generalization to unseen data.
"whyItMatters":"The resource provides a publicly accessible benchmark and baseline models for human‑centered NLG evaluation, enabling the development of systems that better align with human quality perception."
arXiv:2507.01594v2 Announce Type: replace
Abstract: Task-oriented dialogue (ToD) systems aim to help users accomplish goals through natural language interaction. Beyond task success, effective ToD sy...
By Shutong Feng, Hsien-chin Lin, Nurul Lubis, Carel van Niekerk, Michael Heck, Benjamin Ruppik, Renato Vukovic, Milica Ga\v{s}i\'c
UXBench is the first user‑centric benchmark that uses real user feedback to evaluate AI assistants. It comprises three tasks—UX Judge, UX Eval, and UX Recovery—built from 7,400 test instances drawn from over 70,000 interaction logs of a mainstream Chinese AI assistant, covering 8 scenarios and 83 domains. Experiments on 26 state‑of‑the‑art language models reveal how model capability affects user experience and highlight six key findings about user feedback prediction and engagement.
By Mengze Hong, Xia Zeng, Zeyang Lei, Sheng Wang, Chen Jason Zhang, Di Jiang, Taiming Fu, Jinfeng Huang, Mengqiao Liu, Qinghe Chang, Haosheng Zou, Qiongyi Zhou, Sijun He, Simonjmdeng, Minlong Peng, Di Liang
arXiv:2608. 13606v1 Announce Type: new Abstract: The next generation of AI agents is increasingly moving beyond systems that answer isolated questions toward persistent personal assistants that can understand, remember, and continuously learn from users' experiences.
By Xinle Deng, Yida Xue, Xiangyuan Ru, Haoming Xu, Shuofei Qiao, Mengru Wang, Yijun Chen, Buqiang Xu, Chen Jiang, Yuchen Eleanor Jiang, Lizhong Wang, Jianfeng Wang, Li Zeng, Haofen Wang, Guilin Qi, Huajun Chen, Ningyu Zhang
arXiv:2606. 06614v1 Announce Type: cross Abstract: Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data.
By Lechen Zhang, Jiarui Liu, Tal August
Scalaris is a scalable enterprise memory architecture designed to balance retrieval accuracy and context efficiency for conversational AI assistants. It separates user interactions and assistant responses into distinct long‑term and short‑term memory streams, each with specialized representations and ingestion/retrieval pipelines. The system achieves state‑of‑the‑art benchmark scores while reducing retrieved context tokens by about tenfold and improving overall response quality by 36 % in proprietary deployments.
By Devarsh Gandhi, Sarthak Pawar, Hemprasad Badgujar, Vaibhav Rathore, Siddharth Sureka, Neha Bhargava