arXiv AI
Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 18
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
arXiv:2608. 14606v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations ask whether answers look plausible at the individual level.
arXiv Computation and Language
Sep 23
Do Synthetic Personas Predict Real Audience Response? A Sim-to-Real Study Where a No-Persona Baseline Beats Persona-Based Copy Simulation
The study evaluates whether large language models (LLMs) used as synthetic personas can predict real audience responses to marketing copy. Using thousands of headline A/B tests from the Upworthy Research Archive, the authors compare a ten-persona panel grounded in real audience demographics to a no-persona zero‑shot baseline that asks the model for a typical reader’s click likelihood. Results show that the no‑persona baseline outperforms the persona‑based approach, with higher predictive validity and top‑1 accuracy, indicating that forcing the model to role‑play specific personas introduces bias and noise.
arXiv AI
Oct 15
Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas
arXiv:2610.03998v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as synthetic personas representing survey respondents. Their validity as substitutes for particu...
arXiv AI
Jun 19
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
arXiv:2606. 20205v1 Announce Type: new Abstract: Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research.
arXiv AI
Oct 15
Beyond Cultural Persona Bias: Response Geometry as a Measure for Language Models and People
arXiv:2607.02368v4 Announce Type: replace-cross Abstract: Researchers are increasingly prompting language models with cultural personas so that they answer questionnaires as diverse human groups woul...
arXiv Machine Learning
Sep 15
Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging
arXiv:2609.13773v1 Announce Type: new Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may...