arXiv AI

Value-Action Alignment in Large Language Models under Privacy-Prosocial Conflict

arXiv:2601. 03546v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to simulate decision-making tasks involving personal data sharing, where privacy concerns and prosocial motivations can push choices in opposite directions.

arXiv AI
Sep 4

User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios

The study investigates how users perceive the helpfulness and privacy-preservation of large language model (LLM) responses in privacy-sensitive scenarios. Using 94 participants and 90 PrivacyLens scenarios, researchers found that users’ evaluations of identical LLM outputs varied widely, whereas five proxy LLM judges showed high agreement but low correlation with user judgments. The results suggest that proxy LLMs cannot reliably estimate users’ diverse perceptions of utility and privacy, highlighting the need for more user-centered evaluation methods.

By Xiaoyuan Wu, Roshni Kaushik, Wenkai Li, Lujo Bauer, Koichi Onoue
arXiv Computation and Language
4d ago

Evaluating Alignment of Behavioral Dispositions in LLMs

arXiv:2602.11328v2 Announce Type: replace Abstract: As people turn to LLMs for social advice, understanding their behavior in such contexts becomes essential. In this work, we focus on behavioral dis...

By Amir Taubenfeld, Zorik Gekhman, Lior Nezry, Omri Feldman, Natalie Harris, Shashir Reddy, Romina Stella, Ariel Goldstein, Marian Croak, Yossi Matias, Amir Feder
arXiv AI
Aug 7

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

arXiv:2608. 05246v1 Announce Type: new Abstract: Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities.

By Jiahao Zhang, Yongzhi Tong, Zelin Fu, Pengde Zhao, Yanmei Jiang, Jiang Feng, Min Yang
arXiv AI
Sep 4

VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models

The paper introduces VoxPrivacy, a benchmark for assessing interactional privacy in Speech Language Models (SLMs). It evaluates models on a 32‑hour bilingual dataset across three difficulty tiers, revealing that most open‑source SLMs perform near random on conditional privacy decisions and even strong closed‑source systems struggle with proactive privacy inference. The authors also validate these findings on a real‑speech subset and show that fine‑tuning on a 4,000‑hour training set can improve privacy‑preserving capabilities while maintaining robustness.

By Yuxiang Wang, Hongyu Liu, Dekun Chen, Xueyao Zhang, Zhizheng Wu
arXiv Computation and Language
Sep 17

Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialogues

The paper investigates whether large language models (LLMs) can generate synthetic cyberbullying conversations that replicate the social dynamics of real interactions. Using a comprehensive framework, the authors compare authentic dialogues with synthetic ones from GPT, Grok, and LLaMA across structural, linguistic, affective, and temporal dimensions, and conduct human evaluations of realism. Results show that while LLMs preserve high‑level interaction patterns, they systematically distort finer‑grained social phenomena, with model‑specific biases such as GPT’s suppression of harmful content and Grok’s amplification of aggression.

By Arefeh Kazemi, Hamza Qadeer, Sinan Asci, Joachim Wagner, Brian Davis
arXiv AI
Jun 10

IDP-Bench: Benchmarking ability of LLMs to protect personal information in interdependent privacy contexts

arXiv:2606. 09908v1 Announce Type: cross Abstract: Large language models (LLMs) are becoming widely deployed as personal AI assistants with access to sensitive user data, making privacy a major challenge for their design and evaluation.

By Ayana Hussain, Soumya Sharma, Golnoosh Farnadi, Nicholas Vincent, H\'eber Hwang Arcolezi, Ulrich A\"ivodji