The study investigates how users perceive the helpfulness and privacy-preservation of large language model (LLM) responses in privacy-sensitive scenarios. Using 94 participants and 90 PrivacyLens scenarios, researchers found that users’ evaluations of identical LLM outputs varied widely, whereas five proxy LLM judges showed high agreement but low correlation with user judgments. The results suggest that proxy LLMs cannot reliably estimate users’ diverse perceptions of utility and privacy, highlighting the need for more user-centered evaluation methods.
By Xiaoyuan Wu, Roshni Kaushik, Wenkai Li, Lujo Bauer, Koichi Onoue
arXiv:2606. 18062v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used to fulfill users' information needs; users ask LLMs about the weather, pose educational questions, and consult them for legal assistance.
By Hobin Kim, Xiaoyuan Wu, Omer Akgul, Lujo Bauer, Nicolas Christin
arXiv:2603. 23433v3 Announce Type: replace Abstract: AI agents are becoming active decision-makers on the Internet.
By Giulio Frey, Kawin Ethayarajh
arXiv:2602.11328v2 Announce Type: replace
Abstract: As people turn to LLMs for social advice, understanding their behavior in such contexts becomes essential. In this work, we focus on behavioral dis...
By Amir Taubenfeld, Zorik Gekhman, Lior Nezry, Omri Feldman, Natalie Harris, Shashir Reddy, Romina Stella, Ariel Goldstein, Marian Croak, Yossi Matias, Amir Feder
arXiv:2609.18282v1 Announce Type: new
Abstract: Large language models are increasingly used to generate and evaluate online content, yet it remains unclear whether the qualities they associate with h...
By Xinglang Zhang, Yuanmeng Xiang, Yunyao Zhang, Zeliang Chen, Junqing Yu, Zikai Song
arXiv:2608.28833v1 Announce Type: new
Abstract: While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providin...
By Yumeng Wang, Yuchen Wu, Cheng Qian, Zhiyuan Fan, Hyeonjeong Ha, Shujin Wu, Jiayu Liu, Heng Ji, Ge Wang
arXiv:2606. 05256v1 Announce Type: new Abstract: This study analyzes a publicly released dataset from a discontinued field experiment on Reddit's r/ChangeMyView.
By Kokil Jaidka, Saifuddin Ahmed
arXiv:2601. 03546v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to simulate decision-making tasks involving personal data sharing, where privacy concerns and prosocial motivations can push choices in opposite directions.
By Guanyu Chen, Chenxiao Yu, Xiyang Hu
arXiv:2608. 05246v1 Announce Type: new Abstract: Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities.
By Jiahao Zhang, Yongzhi Tong, Zelin Fu, Pengde Zhao, Yanmei Jiang, Jiang Feng, Min Yang
arXiv:2605.29018v2 Announce Type: replace
Abstract: Although a growing body of research has begun to describe user--LLM interactions, the picture it paints is largely static; little is known about ho...
By Rebecca M. M. Hicke, Kiran Tomlinson
Large language models are increasingly used to generate and evaluate online content, yet it remains unclear whether the qualities they associate with higher engagement match what real users respond to...
PersonaMem-v3 is a benchmark and evaluation harness designed to assess omni-platform personal intelligence for AI agents. It is built from over one million anonymized real-world engagement histories, covering social media, chatbots, calendars, and AI companions, and tracks user preferences and habits over time. The benchmark tests agents on personalization, LLM-powered recommendation, proactiveness, agentic tool use, and geo-temporal reasoning, evaluating their ability to infer holistic user understanding, personalize responses, rerank recommendations, follow user steering, and avoid inappropriate personalization.
By Bowen Jiang, Yuan Yuan, Zhuoqun Hao, Yuchen Liu, Maohao Shen, Sihao Chen, Gregory Wornell, Chris Callison-Burch, Lyle Ungar, Dan Roth, Qi Guo, Xiangjun Fan, Camillo J. Taylor, Hanchao Yu