How Much Human Label Variation Does Formal Semantic Structure Explain? Group-Level Effects and Item-Level Ceilings in NLI
Read the original on arXiv Computation and Language →The study investigates whether formal semantic structures—such as negation, quantification, and monotonicity—account for human label variation in natural language inference (NLI). Using the ChaosNLI dataset, where each SNLI and MNLI item is annotated by 100 annotators, the authors apply a rule-based monotonicity tagger and validate it manually. Results show that at the group level, non‑upward‑monotone hypotheses elicit slightly more disagreement, but this effect is sensitive to tagger errors; at the item level, formal structure explains only a few percent of the variation and fails to identify highly disputed items. Thus, formal semantic structure contributes a small, bounded amount to disagreement in NLI annotations.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.