arXiv Computation and Language

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

arXiv AI
Aug 12

UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs

arXiv:2608. 10042v1 Announce Type: cross Abstract: Tool-use LLMs are increasingly asked to act on users' behalf, but existing benchmarks usually focus on profile recall, style imitation, generic tool use, or response-level personalization.

By Xuexiong Yin, Zechuan Chen, Yongsen Zheng, Yuxiang Zhang, Jingyuan Yang, Bin Wang, Yubin Wang, Keze Wang
arXiv Computation and Language
Sep 4

Bounded Personas Match Retrieval on Classification but Not Regression for a Frozen Agent

The paper introduces PersonaLink, a training‑free method that distills a user’s interaction history into a bounded three‑field persona and iteratively refines it by self‑evaluating a frozen 7B language model on held‑out labeled data. Each refinement rewrites the persona only if it does not regress on that slice, ensuring the persona remains bounded and query‑independent. On a 200‑user news categorization task (LaMP‑2), PersonaLink achieves 0.745–0.755 accuracy, statistically indistinguishable from BM25 retrieval’s 0.760–0.765 accuracy, demonstrating that distilled personas can match retrieval for classification but not for regression tasks.

By JaeHa Yoon, Minjun Park, Seoyeon Kim, Jiwoo Lee, Hyunwoo Choi, Dohyun Kang
arXiv Computation and Language
Sep 14

Creating an Atomic User Model for Personality-Aware Large Language Model Interaction

The paper introduces the Atomic User Model (AUM), a structured representation of a user’s personality that separates a stable identity nucleus from four interpretable shells—psychological, cognitive & experiential, behavioural, and social—along with cross-shell entries for conflict and authenticity. It proposes using AUM as a retrieval index rather than a prompt prefix, enabling a task‑specific, budgeted retrieval of relevant fields at generation time. Experiments with simulated participants show that retrieving eight AUM fields improves style fidelity, preference accuracy, and user voice identification compared to flat preference notes, especially benefiting users whose default assistant performs poorly.

By B. Sankar, Deepthika S, Pawni Yadav, Amogh A S
arXiv Computation and Language
Sep 7

Controlling and Assessing Appropriate Persona Use in LLM-based Dialogue Generation

The paper examines how large language models (LLMs) tend to overuse persona attributes in persona-based dialogue generation, producing unnatural responses. It identifies a systematic bias in LLMs to incorporate all provided persona details and shows that current metrics cannot assess contextual appropriateness. To address this, the authors introduce Self-CONtrastive Persona Overuse Suppression (SCONPOS), which intervenes in the prompt encoding stage to reduce overuse, and propose the Persona Appropriateness Score (PAS), a new metric that penalizes both overuse and underuse of persona attributes.

By Jongkyung Shin, Inkyu Lee, Chiehyeon Lim
arXiv AI
4d ago

SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems

SimInter is a pipeline that enriches user profiles for conversational recommender systems by conducting synthetic interviews between two large language model agents, grounded in users’ statistical profiles and historical conversational evidence. An audit agent flags potential fabrication and stereotyping at each turn before enrichment. In experiments on movie recommendation, SimInter significantly increases the diversity of expressed preference aspects and improves simulator performance with two recommenders, with high human agreement on enriched attributes and auditor decisions.

By Erfan Shahabi, Amirhossein Roshandel, Mohammad Sadegh Sadeghi, Azadeh Shakery
arXiv Computation and Language
Sep 10

HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization

HyperTrace is a training‑free framework that personalizes large language models by tracing latent user preferences online. It maintains interpretable natural‑language hypotheses about short‑term intent and long‑term preferences, updating them with an SMC‑style reweighting process driven by an LLM‑based surrogate choice model. Experiments on PRISM and PersonaMem‑v2 demonstrate that HyperTrace improves response alignment, preference prediction, and profile consistency compared to strong online baselines.

By Jianzhi Shen, Keyu Mao, Minghao Shao, Chuanyang Jin, Yusong Wang, Ailiang Lin, Kotaro Funakoshi, Manabu Okumura, Tianmin Shu, Muhammad Shafique