arXiv AI
Oct 31

SocialArena: Evaluating Social Intelligence in Large Language Models

SocialArena is a benchmark designed to evaluate the social intelligence of large language models (LLMs) across three dimensions: role‑playing ability, mental state reasoning, and behavioral reasoning. It introduces a role‑playing dialogue paradigm with explicit character thinking to assess these capabilities in realistic human interactions, and a U‑TRIMF framework to synthesize diverse social contexts. The evaluation relies on a human‑annotated arena‑style protocol, supported by a $132.4K investment in recruiting and training annotators who compare model responses and provide preference labels and critiques.

By Zongqi Wang, Rui Wang, Yuchuan Wu, Hua Yang, Bingxue Qiu, Tianle Gu, Bing Zhao, Yujiu Yang, Jieping Ye, Yongbin Li
arXiv AI
Sep 21

Do Personality-Tuned LLMs Make Better Social Agents?

The paper examines whether fine‑tuning large language models (LLMs) with personality‑labelled data improves their ability to act as socially interactive agents. Two small open‑weight LLMs were fine‑tuned on a corpus of personality‑labelled social media posts and dialogues, and the resulting models were evaluated in various social interaction scenarios by independent LLM judges. The findings show that the fine‑tuned models do not outperform their baseline counterparts in role‑playing personalities, though they offer comparable text quality and increased linguistic diversity for the Qwen models; low inter‑rater agreement limits confidence in the results, suggesting future work should focus on training data quality and domain alignment.

By Tim Krabbe, Xiaodan Shi