arXiv:2607.02368v3 Announce Type: replace-cross Abstract: Language models prompted with cultural personas increasingly stand in for human respondents in cross-cultural research. Their responses separ...
The study audits the cultural values expressed by the decision‑only language model JEV using the 2013 Values Survey Module. By presenting 24 items to JEV under 12 matched Saudi and 12 matched American personas, in both English and Arabic, and across eight request formulations, the researchers found that JEV’s responses were highly repeatable (ICC 0.997) and that persona and language significantly influenced the model’s value profiles. Saudi personas shifted JEV’s answers toward the human Saudi‑US difference—capturing 87 % of the effect in English and 62 % in Arabic—while language, age, and gender also modulated the outcomes.
arXiv:2610.03998v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as synthetic personas representing survey respondents. Their validity as substitutes for particu...
The paper introduces Cultural Divergence Preservation (CDP), a new diagnostic for evaluating whether large language models (LLMs) preserve cross‑country differences when used as synthetic survey respondents. CDP uses a single human calibration to detect cultural flattening (reduced divergence) or caricature (increased divergence) and is shown to vary monotonically with cross‑country divergence, unlike conventional Jensen–Shannon divergence metrics. Experiments across multiple LLM backbones, prompting methods, and survey domains reveal that CDP uncovers systematic discrepancies with traditional fidelity metrics, highlighting that methods favored by those metrics can still produce strong flattening.
arXiv:2606. 20205v1 Announce Type: new Abstract: Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research.
The paper examines how large language models (LLMs) respond to different demographic cues—such as names—when users seek advice, focusing on race and gender in a U.S. context. It finds that using different cues for the same group leads to only partially overlapping changes in model responses, producing inconsistent conclusions about personalization and unstable bias metrics. The authors argue that LLMs react to linguistic signals tied to cues rather than to stable demographic categories, and they call for evaluations that use multiple cues and consider underlying mechanisms.
arXiv:2608. 14606v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations ask whether answers look plausible at the individual level.
arXiv:2501. 02211v3 Announce Type: replace-cross Abstract: Large language models (LLMs) reproduce homogeneity bias -- the tendency to portray marginalized groups as more internally similar than dominant groups -- but whether this bias generalizes across models, is stable under different inference settings, or depends on how group identity is signaled remains unstudied.
arXiv:2501. 02211v2 Announce Type: replace-cross Abstract: Large language models (LLMs) reproduce homogeneity bias -- the tendency to portray marginalized groups as more internally similar than dominant groups -- but whether this bias is stable or an artifact of inference settings has only been studied in single proprietary models.
The study investigates how different prompt components affect language model responses in psychometric tests. By crossing five distinct baseline personas with five variants of each prompt element—persona wording, task instruction, item wording, and option symbol—the authors measure response shifts using the 1‑Wasserstein distance. Their analysis of 13 small open‑weight language models on the Big Five Inventory and Short Dark Triad reveals that task instruction and option symbol changes often cause more variation than paraphrasing the persona or item, with prompt artifacts explaining over 50% of the variation for many items.
arXiv:2609.13773v1 Announce Type: new Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may...
The paper evaluates three open‑weight large language models—Gemma3‑12B (USA), Bielik‑11B‑v3 (Poland), and Qwen3‑4B (China)—against World Values Survey data for 63 demographic personas across three countries, using normalized Wasserstein distance to measure cultural misalignment. Surprisingly, none of the models shows a preference for its home country; Qwen3‑4B, built in China, has the highest misalignment for Chinese respondents. Targeted LoRA fine‑tuning on the five worst‑case personas, with fewer than 1,200 training pairs and under 15 minutes on a single GPU, reduces bias by 16.8% for Bielik‑11B, but the fine‑tuning redistributes bias rather than eliminating it, shifting worst‑case personas from American to Chinese elderly.