arXiv AI

Human Psychometric Questionnaires Mischaracterize LLM Behavior

The paper investigates whether human psychometric questionnaires can reliably characterize large language models (LLMs) in everyday interactions. By comparing eight open‑source LLMs’ value and personality profiles from Likert self‑reports (PVQ‑40/21 and BFI‑44/10) with generation probabilities on value‑laden user queries, the authors find substantial divergence between the two methods. The study shows that questionnaire items contain explicit lexical cues that lead models to respond in socially desirable ways, whereas realistic user queries lack such cues, and demographic persona prompts shift questionnaire responses but not generation outputs, indicating that questionnaire scores overestimate LLMs’ true behavioral tendencies.

arXiv Computation and Language

5d ago

PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs

PACIFIC is a framework that aligns large language model responses with user preferences by leveraging stable Big‑Five personality traits as a latent signal. The authors built a 1,200‑pair dataset covering diverse domains and trait directions, and found that trait‑aligned contexts enable LLMs to achieve near‑ceiling accuracy (up to 99%) in personalized QA. They also introduced a persona‑aware contrastive retriever (PiRAG) that improves label‑free accuracy from 30% to 43% over standard semantic retrieval, highlighting retrieval as the main bottleneck.

By Tianyu Zhao, Siqi Li, Yasser Shoukry, Salma Elmalaki

llms safety

arXiv AI

Jul 31

From Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMs

arXiv:2607. 26853v1 Announce Type: cross Abstract: Human personality theories characterize traits not as isolated attributes captured by a single score, but as stable individual tendencies expressed through the interplay among persons, situations, and behaviors.

By Ruikang Zhang, Shuo Wang, Qi Su

llms

arXiv AI

Aug 19

Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits

The paper introduces the Core Sentiment Inventory (CSI), a new personality trait evaluation tool for large language models (LLMs) that addresses reliability and validity issues found in existing methods like the Big Five Inventory (BFI). CSI is designed specifically for LLMs, supports both English and Chinese, and provides detailed psychological portraits of model behavior. Experiments show that CSI captures nuanced behavioral patterns, improves reliability, and correlates strongly (above 0.85) with real-world LLM outputs.

By Huanhuan Ma, Haisong Gong, Xiaoyuan Yi, Xing Xie, Philip S. Yu, Dongkuan Xu

llms

arXiv AI

Jun 12

Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

arXiv:2606. 12730v1 Announce Type: new Abstract: Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior.

By Rafal Kocielnik, Pengrui Han, Peiyang Song, Myrl G. Marmarelis, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez

llms safety

arXiv AI

Jun 17

Teaching Values to Machines: Simulating Human-Like Behavior in LLMs

arXiv:2605. 30036v2 Announce Type: replace Abstract: Large Language Models (LLMs) demonstrate a remarkable capacity to adopt different personas and roles; however, it remains unclear whether they can manifest behavior that adheres to a coherent, human-like value structure.

By Asaf Yehudai, Naama Rozen, Ariel Gera

llms safety

arXiv AI

Sep 2

Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models

arXiv:2412.12144v5 Announce Type: replace-cross Abstract: Personality assessment through situational judgment tests (SJTs) offers unique advantages over traditional Likert-type self-report scales, ye...

By Chang-Jin Li, Jiyuan Zhang, Yun Tang, Jian Li

llms

arXiv AI

Aug 25

Evaluating the Efficacy of LLMs to Emulate Realistic Human Personalities

arXiv:2402.14879v2 Announce Type: replace-cross Abstract: To enhance immersion and engagement in video games, the design of Affective Non-Player Characters (ANPCs) is a key focus for researchers and...

By Lawrence J. Klinkert, Stephanie Buongiorno, Corey Clark

llms benchmarks safety

arXiv AI

Jul 31

Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

arXiv:2510. 22170v3 Announce Type: replace Abstract: Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation.

By Alexandra Yost, Shreyans Jain, Shivam Raval, Grant Corser, Allen Roush, Nina Xu, Jacqueline Hammack, Ravid Shwartz-Ziv, Amirali Abdullah

llms benchmarks

arXiv AI

Jun 19

Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact

arXiv:2606. 20205v1 Announce Type: new Abstract: Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research.

By Jelena Meyer, David Garcia, Dirk U. Wulff

llms safety

arXiv AI

Aug 12

Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control

arXiv:2608. 10703v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly act in interactive settings where their behavioral styles affect user experience, safety, and downstream decision making.

By Haoze Liu, Run Liu, Haiying Xu, Jiahui Han, Siyuan Fang, Siyu Yan, Huiqi Deng, Guanchu Wang, Na Zou

llms safety

arXiv AI

Aug 18

Beyond Asking: A Pipeline for Personalized Game Generation that Reads Players from Behavior

arXiv:2608. 16196v1 Announce Type: new Abstract: Personalized game generation requires inferring a player's abilities and behavioral style from how they play.

By Yifan Lu, Xiaopeng Yuan, Haohan Wang

llms rag robotics benchmarks

arXiv Computation and Language

Sep 3

When Persona Attributes Improve Population Alignment in Large Language Models

The paper investigates how persona prompting—using short textual descriptions of individuals—to align large language models (LLMs) with human survey responses. It examines the impact of selecting different persona attributes and finds that not all attribute combinations improve performance, suggesting that the variation in human responses to survey questions may explain mixed results. The study evaluates multiple attribute selection methods across four social surveys, two countries, six LLMs, and twenty prediction tasks, offering guidance on when persona prompting is beneficial and which attribute choices are most effective.

By Leon Fr\"ohling, Jens Rupprecht, Markus Strohmaier, Claudia Wagner

llms safety

Related stories