The paper investigates how persona prompting—using short textual descriptions of individuals—to align large language models (LLMs) with human survey responses. It examines the impact of selecting different persona attributes and finds that not all attribute combinations improve performance, suggesting that the variation in human responses to survey questions may explain mixed results. The study evaluates multiple attribute selection methods across four social surveys, two countries, six LLMs, and twenty prediction tasks, offering guidance on when persona prompting is beneficial and which attribute choices are most effective.
By Leon Fr\"ohling, Jens Rupprecht, Markus Strohmaier, Claudia Wagner
arXiv:2602. 14279v2 Announce Type: replace-cross Abstract: Eliciting information to reduce uncertainty about latent group-level properties from surveys and other collective assessments requires allocating limited questioning effort under real costs and missing data.
By Ruomeng Ding, Tianwei Gao, Thomas P. Zollo, Eitan Bachmat, Richard Zemel, Zhun Deng
The paper investigates token‑level certainty as a proxy for correctness in large language models. It finds that certainty better predicts whether a model will answer a question correctly than it does whether a specific response is correct, and that certainty varies by token type and position. The authors show that using certainty early in generation to allocate responses and later to weight votes improves accuracy while dramatically cutting token cost.
By Yunfan Zhou, Ye Zhu, Zhihai Wang, Jianguo Yao, Haibing Guan, Xijun Li
arXiv:2608.22582v1 Announce Type: cross
Abstract: Large-scale population surveys are essential for generating robust social and scientific insights, yet they face significant challenges, including de...
By Julia Romberg, Tobias Gummer, Gabriella Lapesa, Tanja Kunz, Claudia Wagner
arXiv:2605.10659v3 Announce Type: replace
Abstract: Digital personas powered by Large Language Models (LLMs) are increasingly proposed as substitutes for human survey respondents, yet it remains uncl...
By Mumin Jia, Yilin Chen, Divya Sharma, Jairo Diaz-Rodriguez
The paper investigates the mismatch between large language models’ verbalized (knowledge) distributions and their actual output (behavior) distributions when predicting group opinions. It shows that for 37% of World Values Survey questions, the most likely answer under the verbalized distribution differs from that under the log‑probability distribution, indicating a disconnect between what the model knows and what it outputs. To reconcile this, the authors propose a learnable combination of the two distributions that preserves the knowledge shape while aligning with the behavior’s most likely answer, achieving higher behavioral consistency with only a modest increase in distance to human opinion distributions.
By Yang Liu, Masahiro Kaneko, Chenhui Chu
VISTA is a method that steers large language models by inferring a latent user value profile from a few survey answers and applying a hypernetwork to generate user‑specific interventions. It outperforms or matches in‑context profiling on synthetic and real datasets (Food2D, World Values Survey, PrimeX) without adding prompt tokens. VISTA also adaptively selects which survey questions to ask, stopping after only 6% of the questionnaire while preserving 94% of the full‑history benefit, and identifies a clear value axis that accurately positions unseen users.
By Yinuo Xu, Jadie Adams, Brian Hu, Shivani Kumar, David Jurgens, Arslan Basharat
The paper proposes a model-based evaluation framework that merges multidimensional item response theory (IRT) with question context embeddings to predict large language model (LLM) performance on unseen questions. By representing LLMs with latent capability profiles and incorporating question content to inform item characteristics, the approach improves prediction accuracy over model-free baselines in within-scenario settings and offers a richer description of capability variation than unidimensional models. However, the study also finds that this generalizability does not reliably extend to cross-scenario shifts, indicating a key limitation for broader application.
By Ergan Shang, Weijing Tang, Yinqiu He
arXiv:2509. 22888v2 Announce Type: replace Abstract: Standard LLM evaluation practices compress diverse abilities into single scores, obscuring their inherently multidimensional nature.
By Louie Hong Yao, Nicholas Jarvis, Tiffany Zhan, Saptarshi Ghosh, Linfeng Liu, Tianyu Jiang
arXiv:2604. 17267v2 Announce Type: replace Abstract: Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions.
By Zikun Ye, Hema Yoganarasimhan
arXiv:2606. 12754v1 Announce Type: cross Abstract: Are large language models (LLMs) bad at capturing human judgment?
By Danica Dillion, Chen Cecilia Liu, Baihui Wang, Daniele Barolo, Tanmay Rajore, Niket Tandon, Pranathi Ravikumar, Kurt Gray
Silicon sampling creates synthetic survey respondents by having large language models answer questions as people with specified demographics, and these synthetic populations are increasingly replacing real respondents. Current evaluations focus on answer accuracy and distributional matches but do not assess whether the synthetic respondents resemble actual human respondents. The authors introduce the respondent coverage gap (RCG) metric, which measures how far each real respondent is from the nearest synthetic one, and find that all six evaluated LLMs leave human respondents uncovered, performing worse than a baseline that samples answers independently from human marginals.
By Yusuke Kumagae, Masahiro Kaneko