arXiv Computation and Language

AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interaction

arXiv AI
Oct 31

SocialArena: Evaluating Social Intelligence in Large Language Models

SocialArena is a benchmark designed to evaluate the social intelligence of large language models (LLMs) across three dimensions: role‑playing ability, mental state reasoning, and behavioral reasoning. It introduces a role‑playing dialogue paradigm with explicit character thinking to assess these capabilities in realistic human interactions, and a U‑TRIMF framework to synthesize diverse social contexts. The evaluation relies on a human‑annotated arena‑style protocol, supported by a $132.4K investment in recruiting and training annotators who compare model responses and provide preference labels and critiques.

By Zongqi Wang, Rui Wang, Yuchuan Wu, Hua Yang, Bingxue Qiu, Tianle Gu, Bing Zhao, Yujiu Yang, Jieping Ye, Yongbin Li
arXiv AI
Sep 21

Do Personality-Tuned LLMs Make Better Social Agents?

The paper examines whether fine‑tuning large language models (LLMs) with personality‑labelled data improves their ability to act as socially interactive agents. Two small open‑weight LLMs were fine‑tuned on a corpus of personality‑labelled social media posts and dialogues, and the resulting models were evaluated in various social interaction scenarios by independent LLM judges. The findings show that the fine‑tuned models do not outperform their baseline counterparts in role‑playing personalities, though they offer comparable text quality and increased linguistic diversity for the Qwen models; low inter‑rater agreement limits confidence in the results, suggesting future work should focus on training data quality and domain alignment.

By Tim Krabbe, Xiaodan Shi
arXiv Machine Learning
Jul 30

Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinement

arXiv:2607. 26473v1 Announce Type: new Abstract: Personalizing large language models (LLMs) to individual users is essential for improving user experience, yet existing approaches typically rely on explicit preference supervision such as pairwise comparisons or demographic attributes, limiting their applicability in natural interaction settings.

By Haifeng Wu