The paper surveys how annotator disagreement in NLP is increasingly treated as a meaningful signal rather than noise. It offers a domain‑agnostic taxonomy of disagreement sources, a unified framework for modeling approaches, and a review of evaluation metrics for predictive performance and annotator behavior. The authors highlight a shift toward explicit disagreement modeling, structured annotator relationships, and outline future challenges such as integrating multiple variation sources and developing interpretability frameworks.
By Yinuo Xu, David Jurgens
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
By Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger
arXiv:2604. 17289v2 Announce Type: replace Abstract: Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise.
By Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, Ramtin Pedarsani
IDEAlign introduces a new protocol for evaluating the similarity of large language model (LLM) annotations to expert judgments. It uses pick‑the‑odd‑one‑out tasks to capture expert similarity and benchmarks various similarity methods—including text embeddings, topic models, and LLM-as-a-judge—against these human ratings. Applied to educational datasets, the study finds that most metrics miss nuanced expert dimensions, with LLM-as-a-judge performing best yet still insufficient for full expert alignment.
By Hyunji Nam, Lucia Langlois, James Malamut, Mei Tan, Dorottya Demszky
The paper evaluates whether breaking down evaluation tasks into simpler sub‑tasks improves the LLM-as-a-judge (LLMaJ) framework for reference‑free NLG assessment. Systematic comparisons across multiple datasets show no performance advantage for LLMaJ methods that use task decomposition over a fair baseline that does not. The authors attribute previously reported gains to the use of human labels for training rather than to decomposition itself, and note that LLMaJ without decomposition can match human annotators when such labels are available.
By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
The paper explores using LLM-generated annotations to train smaller BERT models for literary sentiment analysis. Five BERT variants were fine‑tuned with ordinal regression on 10,889 sentences labeled by Gemma 4 31B on a ten‑point scale, and evaluated on 1,923 human‑annotated sentences from Hemingway’s *The Old Man and the Sea*. BERT‑large achieved a Kendall’s τ of 0.589, close to the teacher’s 0.604, and BERT‑base outperformed the teacher on non‑neutral sentences with an MAE of 0.726 versus 1.116.
By Artem Suslov, Takehiro Hashimoto