The study investigates how users perceive the helpfulness and privacy-preservation of large language model (LLM) responses in privacy-sensitive scenarios. Using 94 participants and 90 PrivacyLens scenarios, researchers found that users’ evaluations of identical LLM outputs varied widely, whereas five proxy LLM judges showed high agreement but low correlation with user judgments. The results suggest that proxy LLMs cannot reliably estimate users’ diverse perceptions of utility and privacy, highlighting the need for more user-centered evaluation methods.
By Xiaoyuan Wu, Roshni Kaushik, Wenkai Li, Lujo Bauer, Koichi Onoue
arXiv:2606. 18062v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used to fulfill users' information needs; users ask LLMs about the weather, pose educational questions, and consult them for legal assistance.
By Hobin Kim, Xiaoyuan Wu, Omer Akgul, Lujo Bauer, Nicolas Christin
arXiv:2603. 23433v3 Announce Type: replace Abstract: AI agents are becoming active decision-makers on the Internet.
By Giulio Frey, Kawin Ethayarajh
arXiv:2609.18282v1 Announce Type: new
Abstract: Large language models are increasingly used to generate and evaluate online content, yet it remains unclear whether the qualities they associate with h...
By Xinglang Zhang, Yuanmeng Xiang, Yunyao Zhang, Zeliang Chen, Junqing Yu, Zikai Song
arXiv:2608.28833v1 Announce Type: new
Abstract: While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providin...
By Yumeng Wang, Yuchen Wu, Cheng Qian, Zhiyuan Fan, Hyeonjeong Ha, Shujin Wu, Jiayu Liu, Heng Ji, Ge Wang
arXiv:2606. 05256v1 Announce Type: new Abstract: This study analyzes a publicly released dataset from a discontinued field experiment on Reddit's r/ChangeMyView.
By Kokil Jaidka, Saifuddin Ahmed
arXiv:2601. 03546v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to simulate decision-making tasks involving personal data sharing, where privacy concerns and prosocial motivations can push choices in opposite directions.
By Guanyu Chen, Chenxiao Yu, Xiyang Hu
arXiv:2608. 05246v1 Announce Type: new Abstract: Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities.
By Jiahao Zhang, Yongzhi Tong, Zelin Fu, Pengde Zhao, Yanmei Jiang, Jiang Feng, Min Yang
arXiv:2605.29018v2 Announce Type: replace
Abstract: Although a growing body of research has begun to describe user--LLM interactions, the picture it paints is largely static; little is known about ho...
By Rebecca M. M. Hicke, Kiran Tomlinson
PersonaMem-v3 is a benchmark and evaluation harness designed to assess omni-platform personal intelligence for AI agents. It is built from over one million anonymized real-world engagement histories, covering social media, chatbots, calendars, and AI companions, and tracks user preferences and habits over time. The benchmark tests agents on personalization, LLM-powered recommendation, proactiveness, agentic tool use, and geo-temporal reasoning, evaluating their ability to infer holistic user understanding, personalize responses, rerank recommendations, follow user steering, and avoid inappropriate personalization.
By Bowen Jiang, Yuan Yuan, Zhuoqun Hao, Yuchen Liu, Maohao Shen, Sihao Chen, Gregory Wornell, Chris Callison-Burch, Lyle Ungar, Dan Roth, Qi Guo, Xiangjun Fan, Camillo J. Taylor, Hanchao Yu
The paper introduces ASURRE, a benchmark dataset for detecting AI‑assisted responses in online surveys. It evaluates how different LLM usage strategies—ranging from full generation to persona‑grounded agentic completion—affect the performance of existing machine‑generated text detectors. The study finds that while naive AI usage is easily detected, more sophisticated persona‑grounded agents approach chance performance, yet still leave identifiable behavioural traces that can be aggregated to improve detection.
By Qizhou Wang, Bogdan Mamaev, Christopher Leckie
arXiv:2510. 04465v3 Announce Type: replace-cross Abstract: LLM agents require personal information for personalization in order to effectively act on users' behalf, but this raises privacy concerns that can discourage data sharing, limiting both the autonomy levels at which agents can operate and the effectiveness of personalization.
By Zhiping Zhang, Yi Evie Zhang, Freda Shi, Tianshi Li