The paper investigates the patterns of disagreement in emotion annotation by analyzing multilabel annotations from 1,677 sentence-level units of Hemingway’s *The Old Man and the Sea* and comparing them with individual-level annotations from BRIGHTER and GoEmotions. It identifies three recurring disagreement structures: systematic overlap of emotion categories, differences in label breadth and boundary treatment, and sparse participation or abstention that mimic disagreement. The study shows that hard aggregation obscures these patterns, while annotator vote-share representations preserve only marginal frequencies, and it proposes a vocabulary for describing disagreement structure.
arXiv:2608. 15619v1 Announce Type: new Abstract: Emotion recognition from text keeps improving on benchmarks, yet whether an accuracy ceiling has been reached is seldom asked with discipline.
arXiv:2606. 28772v1 Announce Type: cross Abstract: Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training.
The paper surveys how annotator disagreement in NLP is increasingly treated as a meaningful signal rather than noise. It offers a domain‑agnostic taxonomy of disagreement sources, a unified framework for modeling approaches, and a review of evaluation metrics for predictive performance and annotator behavior. The authors highlight a shift toward explicit disagreement modeling, structured annotator relationships, and outline future challenges such as integrating multiple variation sources and developing interpretability frameworks.
arXiv:2609.05806v1 Announce Type: new Abstract: Emotion Recognition in Conversations (ERC) aims to identify speakers' emotions in multi-turn dialogue. Accurate emotion recognition can support a wide...
arXiv:2403.17612v3 Announce Type: replace Abstract: Labeling corpora constitutes a bottleneck to create models for new tasks or domains. Large language models mitigate the issue with automatic corpus...
Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjective. We present a resource of 1,482 tweets from politically controversial discourse, annotated by five annotators for the presence of enthymemes and their argument structure, designed to study label variation.
arXiv:2606.12186v2 Announce Type: replace Abstract: Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjec...
The paper introduces DNE‑ElecDeb, an enriched version of the USElecDeb dataset that annotates Debate Named Entities (DNEs) in both argumentative and non‑argumentative spans, and defines Debate Named Entity Recognition (DNER) as a new task. It proposes Joint Argument and Entity Tagging (JAET), a generative framework that fine‑tunes decoder‑only LLMs to insert inline argument and entity tags into debate turns while preserving the original transcript. JAET achieves significant improvements in joint AM+DNER performance (+27.3% relative F1 in the untyped setting and +41.9% in the typed setting) over sequential pipelines, and these gains generalize to Persuasive Essays (+26.6% and +52.7%).
The paper introduces AXLE, a pre‑annotation stage that uses a scenario vignette and guided conversation to generate a first‑person micro‑narrative for annotators. In a 2x2 study with 104 annotators on ambiguous hate‑speech items, AXLE increased identity resonance, sensitivity to harm, label entropy, and reduced inter‑annotator agreement, suggesting it surfaces plural perspectives rather than forcing consensus.
Political debates are often analyzed through Argument Mining (AM) to investigate the key arguments that drive them. However, political arguments are rarely interpretable from argumentative spans alone...
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
The study investigates whether formal semantic structures—such as negation, quantification, and monotonicity—account for human label variation in natural language inference (NLI). Using the ChaosNLI dataset, where each SNLI and MNLI item is annotated by 100 annotators, the authors apply a rule-based monotonicity tagger and validate it manually. Results show that at the group level, non‑upward‑monotone hypotheses elicit slightly more disagreement, but this effect is sensitive to tagger errors; at the item level, formal structure explains only a few percent of the variation and fails to identify highly disputed items. Thus, formal semantic structure contributes a small, bounded amount to disagreement in NLI annotations.