arXiv AI

Large Language Models as Digital Twins: a Distribution-Aware Questionnaire-Based Evaluation

The paper examines whether large language models can emulate a specific individual's responses to the BFI-2-60 Big Five personality questionnaire using their social‑media posts. GPT‑4, Gemma, Llama 3.1, and Mistral were evaluated on the MyPersonality dataset, comparing predicted trait scores to actual scores and to general population baselines. Results show that user text does not reliably improve predictions; models perform best near the trait median, struggle at extremes, and GPT‑4 best translates explicit trait profiles into consistent responses, though overall consistency with individual behavior is lacking.

arXiv AI
Aug 19

Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits

The paper introduces the Core Sentiment Inventory (CSI), a new personality trait evaluation tool for large language models (LLMs) that addresses reliability and validity issues found in existing methods like the Big Five Inventory (BFI). CSI is designed specifically for LLMs, supports both English and Chinese, and provides detailed psychological portraits of model behavior. Experiments show that CSI captures nuanced behavioral patterns, improves reliability, and correlates strongly (above 0.85) with real-world LLM outputs.

By Huanhuan Ma, Haisong Gong, Xiaoyuan Yi, Xing Xie, Philip S. Yu, Dongkuan Xu
arXiv AI
Jul 10

Persona Cartography: Charting Language Model Personality Traits in Weight Space

arXiv:2607. 07916v1 Announce Type: new Abstract: Large language models exhibit recurring behavioural patterns -- personas -- that shape generalisation and safety, but we lack reliable tools for decomposing, measuring, and controlling them.

By Luke Baines, Anton Gonzalvez Hawthorne, Mariia Koroliuk, Irakli Shalibashvili, Cl\'ement Dumas, Konstantinos Voudouris, David Demitri Africa
arXiv Computation and Language
Oct 23

On the Behavioral Traits of LLM Agents

The paper introduces A-B-D, a method that infers personality traits of large language model agents from their behavioral data, combining functional actions and linguistic outputs across 345,667 real-world trajectories of 80 models. After filtering for stability and discriminability, 79 features are reduced to six reliable factors—two functional and four linguistic—revealing differences such as Claude‑Sonnet‑4.6’s planfulness versus GPT‑5.5’s low energy. The study also demonstrates a weak correlation between these inferred traits and agents’ self-reported Big Five scores, highlighting a knowledge‑action gap.

By Haokai Zhao, Jie Gao, Yunze Xiao, Xintao Wang, Weihao Xuan, Aditya Joshi, Mark Dredze, Jen-tse Huang
arXiv AI
Sep 4

Human Psychometric Questionnaires Mischaracterize LLM Behavior

The paper investigates whether human psychometric questionnaires can reliably characterize large language models (LLMs) in everyday interactions. By comparing eight open‑source LLMs’ value and personality profiles from Likert self‑reports (PVQ‑40/21 and BFI‑44/10) with generation probabilities on value‑laden user queries, the authors find substantial divergence between the two methods. The study shows that questionnaire items contain explicit lexical cues that lead models to respond in socially desirable ways, whereas realistic user queries lack such cues, and demographic persona prompts shift questionnaire responses but not generation outputs, indicating that questionnaire scores overestimate LLMs’ true behavioral tendencies.

By Woojung Song, Dongmin Choi, Yoonah Park, Jongwook Han, Eun-Ju Lee, Yohan Jo
arXiv Computation and Language
Oct 28

Personality Latents: Conditioning Frozen Language Models on Individuals from Their Own Writing

The paper introduces Personality Latents (PL), a method that learns a latent representation for each individual directly from their own writing without requiring explicit personality labels or written profiles. These latents are injected into a frozen language model as a soft prefix, enabling the model to generate responses that reflect the individual's personality. Experiments show that PL outperforms persona embeddings and prompt-based baselines on MBTI axis prediction, and a latent learned with Qwen2.5-3B-Instruct can transfer personality to Llama‑3.1‑8B-Instruct via a new adapter.

By Gunhee Cho, Yun-Gyung Cheong