Large language models (LLMs) are increasingly used to annotate datasets for training smaller, task‑specialized models such as named entity recognition (NER). However, current fine‑tuning processes ignore the annotation noise introduced by LLMs, leading to degraded performance, and existing noise‑robust losses fail to handle the heterogeneous nature of NER noise (e.g., missing mentions vs. type errors). The authors propose error‑type‑aware loss reweighting, which applies separate reweighting rules for different erroneous token types, improving F1 scores by 0.8–2.0 percentage points on average and up to 4.6 points on Wikigold at 24.1% noise.
By Elena Merdjanovska, Jonas Golde, Alan Akbik
arXiv:2606. 07630v1 Announce Type: cross Abstract: Real-world datasets across image and text domains are often characterized by skewed class distributions and noisy annotations, which jointly degrade model performance, particularly on minority classes.
By Jiancheng Zhang, Meiqing Li, Qi Zhang, Yinglun Zhu
arXiv:2405. 03386v2 Announce Type: replace Abstract: Training with noisy class labels impairs neural networks' generalization performance.
By Marek Herde, Lukas L\"uhrs, Denis Huseljic, Bernhard Sick
arXiv:2512.12870v2 Announce Type: replace-cross
Abstract: Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are of...
By Pouya Ahadi, Blair Winograd, Camille Zaug, Karunesh Arora, Lijun Wang, Kamran Paynabar
The paper demonstrates that object detection benchmarks suffer from incomplete annotations, with re-annotation of COCO, Pascal VOC, Cityscapes, and KITTI revealing up to a 60% increase in detected objects, especially small, occluded, or densely packed instances. The authors propose a scalable annotation pipeline that uses multiple annotators per object to capture uncertainty and improve recall, and they introduce two new large-scale benchmarks: an uncertainty-aware detection benchmark and a label error detection benchmark based on real errors. Their findings show that benchmark performance is highly sensitive to annotation quality, yet model rankings remain largely unchanged, highlighting the need for uncertainty-aware evaluation to better reflect real-world ambiguity.
By Sarina Penquitt, Jonathan Klees, Antonia van Betteray, Parssa Jashnieh, Peter Stehr, Matthias Rottmann, Lars Schmarje
arXiv:2607. 18465v1 Announce Type: new Abstract: Crowdsourced labeling provides valuable labeled data for domains across natural language processing, computer vision, and video.
By Ju Chen, Sijia Xu, Jun Feng, Zhiqiang Gao, Zhengyi Yang
Loss-Based Active Learning for Neural Abstractive Summarization proposes LOBSTER, an active learning framework that selects unlabeled documents similar to the model’s high‑loss training examples to correct specific weaknesses. The method is tailored for abstractive summarization, addressing instability and computational bottlenecks seen in prior work. Experiments on three benchmark datasets and two backbone models show that LOBSTER matches or surpasses state‑of‑the‑art performance while speeding up query selection by up to 665×.
By Michail Ioannou, Tatiana Passali, George Michalopoulos, Grigorios Tsoumakas
arXiv:2512. 10244v2 Announce Type: replace-cross Abstract: Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones.
By Tian Liu, Anwesha Basu, James Caverlee, Shu Kong
Crowdsourced labeling provides valuable labeled data for domains across natural language processing, computer vision, and video. Label aggregation aims to infer latent true labels from noisy and biased annotations, with the key lying in annotator reliability estimation.
arXiv:2604. 13899v3 Announce Type: replace-cross Abstract: Instruction-tuned LLMs can annotate thousands of instances at low cost.
By Ahmad Dawar Hakimi, Lea Hirlimann, Isabelle Augenstein, Hinrich Sch\"utze
arXiv:2603. 09692v2 Announce Type: replace-cross Abstract: Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains.
By Davit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich, Ido Hakimi, Barna P\'asztor, Andreas Krause
The paper introduces Active Testing, a framework that selects the most informative test samples for annotation in NLP, aiming to reduce human effort while accurately estimating model performance. Experiments across 18 datasets and 4 embedding strategies show up to 95% annotation savings with less than 1% loss in performance estimation accuracy. The authors also propose an adaptive stopping criterion to determine the optimal number of samples without a predefined budget.
By Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu, Fabrizio Silvestri, Amin Mantrach