arXiv Machine Learning

Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

arXiv Computation and Language

2d ago

Item-Mean Surrogates: Why Richer Personas Fail to Deliver Individual Fidelity in LLMs as Human Surrogates

The study investigates whether large language models (LLMs) can accurately simulate individual human responses by testing the assumption that richer personas improve prediction fidelity. Across four public datasets, researchers found that LLMs largely predict item means but fail to capture individual deviations, particularly the stable, person‑by‑item signal that accounts for most variance. Even with fine‑tuning and varied models, LLMs recover only a small fraction of this nuanced signal, highlighting limitations in using them as precise human surrogates.

By Daehwan Ahn, Chengfeng Mao, Dokyun Lee

llms fine-tuning

arXiv AI

Oct 15

Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas

arXiv:2610.03998v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as synthetic personas representing survey respondents. Their validity as substitutes for particu...

By Khashayar Pourtaheri, Ahmad Zareei

llms benchmarks safety

arXiv AI

Jun 17

Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference

arXiv:2606. 17165v1 Announce Type: cross Abstract: Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost.

By Joel Persson, M{\aa}rten Schultzberg, Sebastian Ankargren

llms safety

arXiv Computation and Language

Oct 15

When Can Digital Personas Reliably Approximate Human Survey Findings?

arXiv:2605.10659v3 Announce Type: replace Abstract: Digital personas powered by Large Language Models (LLMs) are increasingly proposed as substitutes for human survey respondents, yet it remains uncl...

By Mumin Jia, Yilin Chen, Divya Sharma, Jairo Diaz-Rodriguez

llms rag safety

arXiv Computation and Language

Sep 23

Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation

The study evaluates whether large language models (LLMs) used as synthetic personas can predict real audience responses to marketing copy. Using thousands of headline A/B tests from the Upworthy Research Archive, the authors compare a ten-persona panel grounded in real audience demographics to a no-persona zero‑shot baseline that asks the model for a typical reader’s click likelihood. Results show that the no‑persona baseline outperforms the persona‑based approach, with higher predictive validity and top‑1 accuracy, indicating that forcing the model to role‑play specific personas introduces bias and noise.

By Alexandre Cristov\~ao Maiorano

llms robotics safety

arXiv AI

Jul 8

An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

arXiv:2606. 09843v3 Announce Type: replace-cross Abstract: Large language models (LLMs) give stable answers to personality questionnaires, yet these self-reports fail to predict how the models behave.

By Juan Manuel Contreras

llms safety

arXiv AI

Aug 18

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

arXiv:2608. 14606v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations ask whether answers look plausible at the individual level.

By Mantas Lukauskas, Viktorija \v{S}arkauskait\.e

llms benchmarks

arXiv AI

Jul 31

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

arXiv:2607. 26348v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions.

By Zihan Chen, Di Zhu, Lei Nico Zheng

llms benchmarks

arXiv Computation and Language

Sep 22

Recognition, Simulation, and Refusal: A Contamination-Aware Study of Classic Psychological Effects in LLM Agents

arXiv:2609.22090v1 Announce Type: new Abstract: An LLM producing the response pattern associated with a human psychological effect is not the same claim as the LLM possessing that bias. We present Ps...

By Joy Bose

llms agents robotics benchmarks safety

arXiv AI

Oct 15

Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

arXiv:2610.14912v1 Announce Type: new Abstract: Language models are increasingly used to create synthetic survey samples, also called silicon samples. Decision models are a new class of language mode...

By Mantas Lukauskas, Viktorija \v{S}arkauskait\.e

llms

arXiv Computation and Language

Sep 10

ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation

arXiv:2605.29791v2 Announce Type: replace Abstract: While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions...

By Yutong Yang, Chenxi Miao, Weikang Li, Yunfang Wu

llms benchmarks safety

arXiv AI

Aug 6

MatrAIx: Simulating the World with 8.3 Billion Persona Agents

arXiv:2608. 04205v1 Announce Type: new Abstract: Human evaluation of AI systems and digital products is costly, slow, and difficult to scale.

By Xiaomin Li, Yuexing Hao, Jianheng Hou, Jintao Huang, Qianfeng Wen, Shirley Huang, Yifan Liu, Xiaoyi Liu, Yilan Fan, Yijun Wang, Koutian Wu, Ruoqi Gao, Muhammad Ahmed Mohsin, Jing Tang, Brihi Joshi, Heming Liu, Zheyuan Deng, Zonglin Di, Sankalp Jajee, Jiuyao Lu, Zhiwei Zhang, Saksham Kapoor, Ishan Gupta, Yunhan Zhao, Chanwoo Park, Yucheng Lu, Bing Hu, Weihang Xiao, Aravind Mohan, Hanwen Xing, Runyu Zhang, Mihir Kulshreshtha, Yuanda Xu, Qianyu Zhu, Dianzhuo Wang, Yuxin Xiao, Bowen Jiang, Yongye Su, Wenhao Chai, Zuxin Liu, Lawrence Yunliang Chen, Xuandong Zhao, Ethan Ye, Shivam Patel, Jason Xie, Alex Martin Richmond, Weixiang Ding, Emre Okcular, Diya Mathew, Ziheng Wang, Rana M. Shahroz Khan, Zhejian Peng, Fang Wu, Fan Nie, Xinyang Han, Yubin Kim, Jiawei Zhang, Zhenting Qi, Huangyuan Su, Xu Pan, Abinitha Gourabathina, Hyewon Jeong, Hemanth Neelgund Ramesh, Kumail Alhamoud, Kimia Hamidieh, Zidi Xiong, Samuel Schmidgall, Pengrui Han, Yepeng Huang, Yongheng Wang, Bowen Yang, Alex Gu, Yuchu Wang, Akshay Paruchuri, Brenna Li, Hejie Cui, Jiayuan Ding, Chaosheng Dong, Jiahao Wang, Yixuan He, Chi Wang, Pamela Bhattacharya, Tianyi Peng, Paul Pu Liang, Mitchell Gordon, Yilun Du, Marinka Zitnik, James Zou, Prasanna Tambe, Philip Torr, Emily Fox, Asu Ozdaglar, Dawn Song

llms agents

Related stories