The study investigates whether formal semantic structures—such as negation, quantification, and monotonicity—account for human label variation in natural language inference (NLI). Using the ChaosNLI dataset, where each SNLI and MNLI item is annotated by 100 annotators, the authors apply a rule-based monotonicity tagger and validate it manually. Results show that at the group level, non‑upward‑monotone hypotheses elicit slightly more disagreement, but this effect is sensitive to tagger errors; at the item level, formal structure explains only a few percent of the variation and fails to identify highly disputed items. Thus, formal semantic structure contributes a small, bounded amount to disagreement in NLI annotations.
arXiv:2607.15870v2 Announce Type: replace Abstract: Human label variation in natural language inference is increasingly treated as signal rather than noise, but how much of it formal semantic structu...
arXiv:2607.19231v2 Announce Type: replace Abstract: Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disa...
Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (arXiv:2607.
The paper investigates the patterns of disagreement in emotion annotation by analyzing multilabel annotations from 1,677 sentence-level units of Hemingway’s *The Old Man and the Sea* and comparing them with individual-level annotations from BRIGHTER and GoEmotions. It identifies three recurring disagreement structures: systematic overlap of emotion categories, differences in label breadth and boundary treatment, and sparse participation or abstention that mimic disagreement. The study shows that hard aggregation obscures these patterns, while annotator vote-share representations preserve only marginal frequencies, and it proposes a vocabulary for describing disagreement structure.
arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.
arXiv:2606. 28772v1 Announce Type: cross Abstract: Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training.
arXiv:2609.21277v1 Announce Type: cross Abstract: How many human judgments does a panel of language models represent? The answer depends on what is matched. We audit categorical judge panels against...
Appending a two-word confirmation tag to a decision question -- "Is X the better choice? " versus "X is the better choice, right?
The study evaluates hybrid retrieval, NLI, and open‑source LLMs for scientific claim verification. Using a benchmark of 2,035 claims, the authors compare a DeBERTa‑based NLI baseline with Llama 3.1, Qwen3, and Mistral LLMs, finding that Qwen3 achieves the highest Macro‑F1 score. A conservative fusion rule that labels a claim as Support or Contradict only when both models agree actually degrades performance, and error analysis reveals that disagreements often overturn correct LLM predictions.
arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?
The paper critiques a recent NLI benchmark that tests the imperfective paradox, arguing that the benchmark suffers from conceptual and evaluation mis-specifications, notably Aspectual Reduction and a lack of strict NLI standards. The authors re-evaluate the benchmark, identify mis-specifications, and construct lexically matched minimal pairs to control for lexical variation. Their experiments reveal that models often exhibit a Sufficiency Bias, accept simple‑past hypotheses without affirming culmination, and that prompting interventions shift label decisions without improving true semantic understanding, highlighting additional failure modes such as compositional aspectual classification errors and surface‑form attraction.
arXiv:2606. 05970v1 Announce Type: cross Abstract: Large language models are increasingly used for structured extraction from clinical free-text notes, but the sensitivity of their output to upstream configuration choices is less understood than their accuracy on fixed benchmarks.