← Back to all news
arXiv AI By Xiaoyuan Wu, Roshni Kaushik, Wenkai Li, Lujo Bauer, Koichi Onoue

User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios

Read the original on arXiv AI →

The study investigates how users perceive the helpfulness and privacy-preservation of large language model (LLM) responses in privacy-sensitive scenarios. Using 94 participants and 90 PrivacyLens scenarios, researchers found that users’ evaluations of identical LLM outputs varied widely, whereas five proxy LLM judges showed high agreement but low correlation with user judgments. The results suggest that proxy LLMs cannot reliably estimate users’ diverse perceptions of utility and privacy, highlighting the need for more user-centered evaluation methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 22

Privacy in Context Is Being Overlooked in the Age of General-Purpose LLMs

The paper argues that as large language models (LLMs) expand, privacy concerns grow, yet current approaches tend to seek a single, universal privacy solution. It emphasizes that privacy is inherently contextual, and warns that a unified framework can give a misleading sense of security and ignore fundamental tensions among desired privacy properties. The authors propose focusing on application‑specific deployment, context‑specific evaluation, and improved privacy literacy as key future directions.

By Prakhar Ganesh, Yash More, Marco Romanelli, Ferdinando Fioretto, Golnoosh Farnadi
llmssafety
More like this →
arXiv AI
Sep 4

VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models

The paper introduces VoxPrivacy, a benchmark for assessing interactional privacy in Speech Language Models (SLMs). It evaluates models on a 32‑hour bilingual dataset across three difficulty tiers, revealing that most open‑source SLMs perform near random on conditional privacy decisions and even strong closed‑source systems struggle with proactive privacy inference. The authors also validate these findings on a real‑speech subset and show that fine‑tuning on a 4,000‑hour training set can improve privacy‑preserving capabilities while maintaining robustness.

By Yuxiang Wang, Hongyu Liu, Dekun Chen, Xueyao Zhang, Zhizheng Wu
llmsfine-tuningbenchmarkssafety
More like this →