The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (arXiv:2607.
arXiv:2607.15870v2 Announce Type: replace Abstract: Human label variation in natural language inference is increasingly treated as signal rather than noise, but how much of it formal semantic structu...
The study investigates whether formal semantic structures—such as negation, quantification, and monotonicity—account for human label variation in natural language inference (NLI). Using the ChaosNLI dataset, where each SNLI and MNLI item is annotated by 100 annotators, the authors apply a rule-based monotonicity tagger and validate it manually. Results show that at the group level, non‑upward‑monotone hypotheses elicit slightly more disagreement, but this effect is sensitive to tagger errors; at the item level, formal structure explains only a few percent of the variation and fails to identify highly disputed items. Thus, formal semantic structure contributes a small, bounded amount to disagreement in NLI annotations.
arXiv:2609.21277v1 Announce Type: cross Abstract: How many human judgments does a panel of language models represent? The answer depends on what is matched. We audit categorical judge panels against...
arXiv:2608. 11138v1 Announce Type: cross Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways.
The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.