The paper surveys how annotator disagreement in NLP is increasingly treated as a meaningful signal rather than noise. It offers a domain‑agnostic taxonomy of disagreement sources, a unified framework for modeling approaches, and a review of evaluation metrics for predictive performance and annotator behavior. The authors highlight a shift toward explicit disagreement modeling, structured annotator relationships, and outline future challenges such as integrating multiple variation sources and developing interpretability frameworks.
By Yinuo Xu, David Jurgens
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
By Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger
arXiv:2604. 17289v2 Announce Type: replace Abstract: Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise.
By Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, Ramtin Pedarsani
IDEAlign introduces a new protocol for evaluating the similarity of large language model (LLM) annotations to expert judgments. It uses pick‑the‑odd‑one‑out tasks to capture expert similarity and benchmarks various similarity methods—including text embeddings, topic models, and LLM-as-a-judge—against these human ratings. Applied to educational datasets, the study finds that most metrics miss nuanced expert dimensions, with LLM-as-a-judge performing best yet still insufficient for full expert alignment.
By Hyunji Nam, Lucia Langlois, James Malamut, Mei Tan, Dorottya Demszky
The paper evaluates whether breaking down evaluation tasks into simpler sub‑tasks improves the LLM-as-a-judge (LLMaJ) framework for reference‑free NLG assessment. Systematic comparisons across multiple datasets show no performance advantage for LLMaJ methods that use task decomposition over a fair baseline that does not. The authors attribute previously reported gains to the use of human labels for training rather than to decomposition itself, and note that LLMaJ without decomposition can match human annotators when such labels are available.
By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
The paper explores using LLM-generated annotations to train smaller BERT models for literary sentiment analysis. Five BERT variants were fine‑tuned with ordinal regression on 10,889 sentences labeled by Gemma 4 31B on a ten‑point scale, and evaluated on 1,923 human‑annotated sentences from Hemingway’s *The Old Man and the Sea*. BERT‑large achieved a Kendall’s τ of 0.589, close to the teacher’s 0.604, and BERT‑base outperformed the teacher on non‑neutral sentences with an MAE of 0.726 versus 1.116.
By Artem Suslov, Takehiro Hashimoto
The paper investigates whether adding Abstract Meaning Representation (AMR) data to large language models (LLMs) improves performance on downstream tasks. By reproducing recent studies and applying a consistent hyperparameter protocol, the authors find that text-only baselines match or surpass AMR-augmented models. A perplexity-based probe shows that AMR does not provide LLMs with additional relational knowledge, suggesting no clear benefit from AMR augmentation.
By Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan
arXiv:2403.17612v3 Announce Type: replace
Abstract: Labeling corpora constitutes a bottleneck to create models for new tasks or domains. Large language models mitigate the issue with automatic corpus...
By Christopher Bagdon, Prathamesh Karmalker, Harsha Gurulingappa, Roman Klinger
arXiv:2602. 05493v2 Announce Type: replace-cross Abstract: Data annotation remains a significant bottleneck in the field of humanities and social sciences, particularly for complex linguistic tasks such as metaphor identification.
By Bingru Li
The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by deco...
arXiv:2606. 05308v1 Announce Type: new Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set.
By Abhishek Divekar
arXiv:2603.29396v2 Announce Type: replace
Abstract: Standard evaluations of Large language models (LLMs) focus on task performance, offering limited insight into whether correct behavior reflects app...
By Zo\"e Prins, Samuele Punzo, Frank Wildenburg, Giovanni Cin\`a, Sandro Pezzelle