LLM-based social simulation is a promising complement to traditional methods such as surveys and behavioral experiments. A core question is how to evaluate the fidelity of LLM-simulated human behavior...
The paper introduces NAPHA, a lightweight post‑hoc alignment method that improves large language model (LLM) predictions of human judgment distributions (HJD) by matching LLM output distributions to HJD through entropy‑based class assignment and specialized alignment models. Experiments on five datasets show that while LLMs perform near human‑level on hard‑label tasks, they struggle with soft‑label predictions, and NAPHA consistently enhances soft‑label accuracy, especially on high‑entropy instances. The study also demonstrates that better entropy class prediction can further boost NAPHA’s effectiveness.
By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
The paper investigates the mismatch between large language models’ verbalized (knowledge) distributions and their actual output (behavior) distributions when predicting group opinions. It shows that for 37% of World Values Survey questions, the most likely answer under the verbalized distribution differs from that under the log‑probability distribution, indicating a disconnect between what the model knows and what it outputs. To reconcile this, the authors propose a learnable combination of the two distributions that preserves the knowledge shape while aligning with the behavior’s most likely answer, achieving higher behavioral consistency with only a modest increase in distance to human opinion distributions.
By Yang Liu, Masahiro Kaneko, Chenhui Chu
arXiv:2606. 14199v1 Announce Type: cross Abstract: Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.
By Xuhui Zhou, Weiwei Sun, Weihua Du, Jiarui Liu, Haojia Sun, Qianou Ma, Tongshuang Wu, Yiming Yang, Maarten Sap
The paper investigates whether fine‑tuning large language models (LLMs) on value survey responses can shift their value‑conditioned outputs and downstream behavior. Researchers first estimate baseline value profiles for several open‑source LLMs using the ten Schwartz values, then fine‑tune the models on survey‑style supervision that selectively alters target value ratings while preserving non‑target values. The resulting shifts are evaluated on held‑out survey questions and on out‑of‑domain behavioral proxies, such as a contextual moral‑judgment benchmark from Reddit AITA posts and a text‑based interactive decision‑making benchmark. The study finds that survey‑based supervision can change both survey responses and downstream behavior, but the transfer is strongly model‑ and value‑dependent and influenced by how value descriptions are framed.
By Shangrui Nie, Florian Mai, David Kacz\'er, Charles Welch, Zhixue Zhao, Lucie Flek
arXiv:2511. 14117v2 Announce Type: replace Abstract: Supervised classifiers output a distribution over classes but are typically trained against a single label obtained by collapsing multiple annotators into a majority vote.
By Agamdeep Singh, Ashish Tiwari, Hosein Hasanbeig, Priyanshu Gupta
arXiv:2606. 05376v1 Announce Type: new Abstract: Many human-centered tasks, including natural language inference (NLI) and emotion recognition (ER), have multiple plausible interpretations, leading to label ambiguity and challenging disagreements across human annotators.
By Jingyao Wu, Ashley Wang, Keane Ong, Paul Pu Liang, Rosalind Picard
arXiv:2607. 08493v1 Announce Type: new Abstract: Subjective NLP tasks often exhibit systematic annotator disagreement, requiring models that represent uncertainty rather than collapse it.
By Xia Cui, Ziyi Huang, N. R. Abeynayake
arXiv:2610.19035v1 Announce Type: new
Abstract: Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduce...
By Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal, Zhun Deng
arXiv:2608.30902v1 Announce Type: new
Abstract: Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation imprac...
By Alessio Galatolo, Meriem Beloucif
arXiv:2607. 26348v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions.
By Zihan Chen, Di Zhu, Lei Nico Zheng
The paper introduces Population Fidelity, an evaluation framework for assessing how well large language models (LLMs) represent human population attitudes. It focuses on three dimensions: group-level accuracy, between-group variation, and the structure of that variation. Using the framework, the authors replicate a prior study on machine bias and test cultural fine-tuning, finding that while fine-tuning improves overall alignment, it does not enhance representation of within-population differences.
By Neemias B. da Silva, Martin Lukk, Ali Sutani, Abhishek Moturu, Harris Yang, Daniel Silver, Matt Ratto, Thiago H. Silva