arXiv:2604. 02951v2 Announce Type: replace-cross Abstract: Human data annotation, especially when involving experts, is often treated as an objective reference.
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
The paper investigates which demographic attributes large language models (LLMs) default to when annotating text without explicit demographic cues. By comparing non‑demographic, placebo‑conditioned, and demographic‑conditioned prompts on politeness and offensiveness tasks in the POPQUORN dataset, the authors find that LLMs exhibit notable gender, race, and age influences in their annotations. This contrasts with earlier studies that reported no such effects, highlighting the importance of considering demographic bias in LLM‑based annotation workflows.
The paper introduces AXLE, a pre‑annotation stage that uses a scenario vignette and guided conversation to generate a first‑person micro‑narrative for annotators. In a 2x2 study with 104 annotators on ambiguous hate‑speech items, AXLE increased identity resonance, sensitivity to harm, label entropy, and reduced inter‑annotator agreement, suggesting it surfaces plural perspectives rather than forcing consensus.
arXiv:2603. 13891v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used for automated text annotation in tasks ranging from academic research to content moderation and hiring.
arXiv:2505.21710v2 Announce Type: replace Abstract: This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate...
arXiv:2609.24574v1 Announce Type: new Abstract: Computational social science increasingly relies on large language models for text annotation, and the validity of published findings now rests on the...
arXiv:2606. 06715v1 Announce Type: cross Abstract: We ask whether topic sentiment has a causal effect on perceived political ideology, and whether the answer depends on who assigns the ideology label.
arXiv:2607. 29334v1 Announce Type: cross Abstract: Conversational AI developed by geopolitical rivals reaches citizens worldwide, raising concerns that it could sway public opinion or be rejected as foreign propaganda, with consequences for democratic discourse and information sovereignty.
The study shows that while large language model (LLM) annotations of stakeholder consultation submissions are highly reproducible (intraclass correlations > 0.99), they do not reliably capture the intended construct measured by structured survey responses. Divergence between LLM-inferred and survey measures varies by stakeholder group, with business associations expressing more AI risk concern in text than in surveys, and spatial autocorrelation indicates neighboring European countries share similar text-based stances. Despite these divergences, survey-reported concerns remain strongly linked to support for explainability across all levels of divergence.
arXiv:2608.29803v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs...
arXiv:2609.22133v1 Announce Type: new Abstract: In this paper, we show that LLM and human coding are observationally equivalent in terms of annotation quality: recent LLMs agree with expert coders at...
The paper investigates whether large language models can learn and reproduce annotator‑specific label‑explanation behavior, using two sentence‑pair tasks with four annotators each. It finds that individual annotator patterns are weak at the single‑annotation level but become detectable after reducing input‑content effects and aggregating across annotators. The authors propose cross‑annotator preference optimization (CAPO), which improves upon prompting and supervised fine‑tuning by better capturing annotator‑specific reasoning while maintaining stable attribution.