arXiv Computation and Language
Observational Equivalence of LLM and Human Annotation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
Hugging Face Trending Papers
Jun 10
A Resource for Enthymeme Detection in Controversial Political Discourse
Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjective. We present a resource of 1,482 tweets from politically controversial discourse, annotated by five annotators for the presence of enthymemes and their argument structure, designed to study label variation.
arXiv Computation and Language
Sep 10
A Resource for Enthymeme Detection in Controversial Political Discourse
arXiv:2606.12186v2 Announce Type: replace Abstract: Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjec...
arXiv AI
Jun 17
Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection
arXiv:2604. 13899v3 Announce Type: replace-cross Abstract: Instruction-tuned LLMs can annotate thousands of instances at low cost.
arXiv AI
Sep 4
Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy
The paper examines how Preference Inference (PI) models used in large-scale participatory democracy platforms can alter the perceived consensus and minority support by predicting missing votes. It introduces a collective‑centric evaluation framework that assesses whether inferred votes maintain key properties of the overall preference landscape, rather than focusing solely on individual prediction accuracy. Using the largest multilingual dataset to date—four consultations with over 90,000 participants, 1 million votes, and 22 languages—the study finds that models with similar predictive accuracy can differ markedly in how well they preserve the collective structure, underscoring that accuracy alone is insufficient for evaluating PI in democratic contexts.
arXiv AI
Jul 1
How Human Feedback Shapes AI-generated Community Notes
arXiv:2606. 30905v1 Announce Type: cross Abstract: Community Notes, a bridging-based crowd-sourced fact-checking system, has emerged as a new mechanism for moderating misleading information on social media and has been adopted by major platforms including X, Facebook, Instagram, Threads, and TikTok.
arXiv Machine Learning
Jul 2
REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations
arXiv:2604. 17289v2 Announce Type: replace Abstract: Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise.