The paper investigates whether disagreement among user simulators can flag unreliable estimates of group response proportions and group differences. Using six language‑model simulators on 60 questions across 12 countries, disagreement more accurately predicts invalidity for group differences (AUC 0.87) than for single group proportions (AUC 0.71). The authors find that shared bias among simulators can obscure errors, but correcting for this bias reduces prediction error without altering disagreement signals.
arXiv:2607. 26348v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions.
The study compares human deliberation in Wason selection tasks with large language model (LLM) agent groups that are seeded with participants’ pre-discussion beliefs. Across various scoring definitions, human consensus rates ranged from 24.0% to 57.0%, whereas LLM agents consistently achieved higher consensus, with gaps of 34–44 percentage points in two sensitivity analyses. Even when early stopping was removed or memorizable answers were eliminated, LLM groups still reached near-unanimous agreement, often on incorrect answers, indicating that simulated consensus does not reflect collective accuracy.
arXiv:2606. 28963v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to simulate social survey responses, yet their outputs exhibit systematic biases: marginal distributions are skewed, response variance is poorly calibrated, and predictor-outcome relationships are attenuated.
arXiv:2604. 02458v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to simulate human responses and estimate treatment effect of interventions when real-world experiments are costly or infeasible.
arXiv:2609.25066v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to simulate human survey responses and behavioral reactions, yet unreliable simulations can mislead...
The paper introduces Population Fidelity, an evaluation framework for assessing how well large language models (LLMs) represent human population attitudes. It focuses on three dimensions: group-level accuracy, between-group variation, and the structure of that variation. Using the framework, the authors replicate a prior study on machine bias and test cultural fine-tuning, finding that while fine-tuning improves overall alignment, it does not enhance representation of within-population differences.
The study examines whether asking large language models (LLMs) to predict whether readers would share misinformation adds useful information beyond predicting credibility. Eight LLM versions were audited on 290 synthetic misinformation articles, yielding 1,256 paired survey responses. While credibility scores explained a substantial portion of sharing variation, adding sharing predictions to credibility did not meaningfully reduce error, indicating that sharing answers contain structured variation but lack established incremental validity for human sharing.
arXiv:2607. 20454v1 Announce Type: cross Abstract: All frontier large language models (LLMs) exhibit response drift -- producing outputs that deviate from expert-validated references -- yet the magnitude and structure of this drift remain uncharacterised by systematic human evaluation.
The paper investigates whether language models’ self-reported confidence is meaningful without additional training. By evaluating three training‑free signals—direct verbalization, post‑hoc probability estimates, and agreement across multiple generations—on 100 TriviaQA questions, the authors find that direct verbalization alone achieves high AUROC scores (0.956 and 0.937) for correctness prediction, while agreement-based methods perform noticeably worse. Re‑eliciting confidence for the same answers shows modest score shifts and occasional decision flips, and an audit of biography claims reveals only a small confidence gap between supported and contradicted statements.
arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.
The wide adoption of LLMs across broad NLG applications heightens the importance of providing users with the means to avert errors and hallucinations. Uncertainty quantification is poised to fill that...
arXiv:2510. 12857v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are now widely deployed in user-facing applications, reaching hundreds of millions of users worldwide.