Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (arXiv:2607.
arXiv:2607.15870v2 Announce Type: replace Abstract: Human label variation in natural language inference is increasingly treated as signal rather than noise, but how much of it formal semantic structu...
The study investigates whether formal semantic structures—such as negation, quantification, and monotonicity—account for human label variation in natural language inference (NLI). Using the ChaosNLI dataset, where each SNLI and MNLI item is annotated by 100 annotators, the authors apply a rule-based monotonicity tagger and validate it manually. Results show that at the group level, non‑upward‑monotone hypotheses elicit slightly more disagreement, but this effect is sensitive to tagger errors; at the item level, formal structure explains only a few percent of the variation and fails to identify highly disputed items. Thus, formal semantic structure contributes a small, bounded amount to disagreement in NLI annotations.
arXiv:2609.21277v1 Announce Type: cross Abstract: How many human judgments does a panel of language models represent? The answer depends on what is matched. We audit categorical judge panels against...
arXiv:2608. 11138v1 Announce Type: cross Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways.
The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.
Appending a two-word confirmation tag to a decision question -- "Is X the better choice? " versus "X is the better choice, right?
arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?
arXiv:2606. 28772v1 Announce Type: cross Abstract: Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training.
The paper introduces a two‑layer taxonomy for pluralistic alignment in large language models, distinguishing between Overton, Steerable, and Distributional pluralism and between aggregate, regional, and subgroup human references. It reviews 70 related works, finding most focus on Steerable and Distributional pluralism, with limited Overton pluralism at the aggregate level. The authors also outline evidence types—perspective validity, conditional fidelity, distributional fidelity, support preservation, and uncertainty in subgroup estimates—to evaluate these objectives.
arXiv:2608.23086v1 Announce Type: new Abstract: Black-box large language models need confidence scores that can separate likely-correct from likely-incorrect outputs, enabling systems to prioritize h...
arXiv:2608. 14509v1 Announce Type: new Abstract: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt.