The paper investigates the patterns of disagreement in emotion annotation by analyzing multilabel annotations from 1,677 sentence-level units of Hemingway’s *The Old Man and the Sea* and comparing them with individual-level annotations from BRIGHTER and GoEmotions. It identifies three recurring disagreement structures: systematic overlap of emotion categories, differences in label breadth and boundary treatment, and sparse participation or abstention that mimic disagreement. The study shows that hard aggregation obscures these patterns, while annotator vote-share representations preserve only marginal frequencies, and it proposes a vocabulary for describing disagreement structure.
By Emily \"Ohman, Anna Koufakou, Eugenia Diegoli
Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjective. We present a resource of 1,482 tweets from politically controversial discourse, annotated by five annotators for the presence of enthymemes and their argument structure, designed to study label variation.
arXiv:2606.12186v2 Announce Type: replace
Abstract: Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjec...
By Martial Pastor, Nelleke Oostdijk
The paper investigates how perspectivist NLP models perform when predicting annotations from different annotators, focusing on those who diverge from the majority or provide fewer annotations. It compiles the largest set of perspectivist datasets with ordinal labels and evaluates various algorithms across them. The study finds that models tend to perform better on annotators who agree with others, while the rate at which annotators provide responses does not significantly affect performance.
By London Lowmanstone IV, Vaagishwaran Sivakumar, Anwesha Samaddar, Akul Mundada, Harnoor Kaur, Maria Gini
arXiv:2606. 28772v1 Announce Type: cross Abstract: Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training.
By Joshua Muhumuza, Joab Ezra Agaba, Mercy Amiyo
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
By Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger