The paper investigates the patterns of disagreement in emotion annotation by analyzing multilabel annotations from 1,677 sentence-level units of Hemingway’s *The Old Man and the Sea* and comparing them with individual-level annotations from BRIGHTER and GoEmotions. It identifies three recurring disagreement structures: systematic overlap of emotion categories, differences in label breadth and boundary treatment, and sparse participation or abstention that mimic disagreement. The study shows that hard aggregation obscures these patterns, while annotator vote-share representations preserve only marginal frequencies, and it proposes a vocabulary for describing disagreement structure.
By Emily \"Ohman, Anna Koufakou, Eugenia Diegoli
Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjective. We present a resource of 1,482 tweets from politically controversial discourse, annotated by five annotators for the presence of enthymemes and their argument structure, designed to study label variation.
arXiv:2606.12186v2 Announce Type: replace
Abstract: Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjec...
By Martial Pastor, Nelleke Oostdijk
The paper investigates how perspectivist NLP models perform when predicting annotations from different annotators, focusing on those who diverge from the majority or provide fewer annotations. It compiles the largest set of perspectivist datasets with ordinal labels and evaluates various algorithms across them. The study finds that models tend to perform better on annotators who agree with others, while the rate at which annotators provide responses does not significantly affect performance.
By London Lowmanstone IV, Vaagishwaran Sivakumar, Anwesha Samaddar, Akul Mundada, Harnoor Kaur, Maria Gini
arXiv:2606. 28772v1 Announce Type: cross Abstract: Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training.
By Joshua Muhumuza, Joab Ezra Agaba, Mercy Amiyo
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
By Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger
arXiv:2601.06631v2 Announce Type: replace
Abstract: Building NLP systems for subjective tasks requires one to ensure their alignment to contrasting human values. We propose the MultiCalibrated Subjec...
By Mohammed Fayiz Parappan, Ricardo Henao
The paper investigates how to detect enthymemes—arguments with unstated premises or conclusions—in controversial political tweets about vaccination and immigration. Despite extensive fine‑tuning, models achieve an F1 score of only about 0.67, prompting the authors to propose new evaluation metrics that compare predictions to the full distribution of annotator votes and a model that directly predicts this distribution. The study also shows that models rely on explicit argumentative structure, often missing implicit links that readers must infer, raising concerns for content moderation systems that depend on majority labels.
By Martial Pastor, Nelleke Oostdijk
arXiv:2502. 08266v3 Announce Type: replace-cross Abstract: Hate speech detection is a crucial task, especially on social media where harmful content can spread quickly.
By Somaiyeh Dehghan, Mehmet Umut Sen, Berrin Yanikoglu
arXiv:2606. 26437v1 Announce Type: cross Abstract: Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to capture when both supporting and contradicting evidence coexist.
By Siyi Liu, Aaron Halfaker, Dan Roth, Patrick Xia
arXiv:2512. 06227v3 Announce Type: replace-cross Abstract: Real-world indicators play an important role in many Natural Language Processing (NLP) applications, such as life events for mental health analysis and risky behaviours for online safety, yet labelling such information is often costly and/or difficult due to its multi-label and dynamic nature.
By Junyu Mao, Anthony Hills, Talia Tseriotou, Maria Liakata, Aya Shamir, Dan Sayda, Dana Atzil-Slonim, Natalie Djohari, Pamela Ugwudike, Mahesan Niranjan, Stuart E. Middleton
The paper introduces AXLE, a pre‑annotation stage that uses a scenario vignette and guided conversation to generate a first‑person micro‑narrative for annotators. In a 2x2 study with 104 annotators on ambiguous hate‑speech items, AXLE increased identity resonance, sensitivity to harm, label entropy, and reduced inter‑annotator agreement, suggesting it surfaces plural perspectives rather than forcing consensus.
By Sheza Munir, Ramaravind Kommiya Mothilal, Syed Ishtiaque Ahmed