The paper investigates how benchmark contamination—leakage of test items into training data—affects large language model (LLM) leaderboards. By comparing original test items with semantically equivalent paraphrases, the authors measure contamination as a violation of anchor-item invariance and find that it inflates absolute scores but rarely changes model rankings. Across 47 public models and 74 finetuned models on four benchmarks, the rank correlation between standard and paraphrase-controlled leaderboards is 0.997, with only a handful of cases showing differential contamination that could alter rankings.
By Xingyao Xiao (Stanford University), Yihong Cheng (City University of Macau)
arXiv:2610.14015v1 Announce Type: cross
Abstract: Learning-to-rank (LTR) models can align rankings with undesignated signals when those signals appear as features or leak into training labels. We com...
By Vladyslav Shevchuk, Carlos Uziel P\'erez Malla, Philipp Wissgott
The paper presents a high‑dimensional theory of learning with data pruning via weighted empirical risk minimization. It analyzes pruning along a one‑dimensional subspace and derives limiting test‑error curves as input dimension and sample size grow with a fixed ratio. The authors show that, in the high‑sample regime, the optimal strategy is to keep only the 2N hardest examples, achieving a Bayes‑optimal error scaling of order N/P instead of the usual sqrt(N/P).
By Elvis Dohmatob
arXiv:2610.01028v1 Announce Type: cross
Abstract: Machine learning models often suffer performance degradation under subpopulation shift, particularly when spurious correlations cause models to rely...
By Sung Ho Jo, Seonghwi Kim, Wonsang Yun, Minwoo Chae
We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists.
arXiv:2607. 24622v1 Announce Type: cross Abstract: We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones.
By Gabriel Singer, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis, Argyris Kalogeratos
Despite a surge of empirical interest in the ``unreasonable effectiveness'' of drastic data pruning for AI systems, a unified theoretical understanding of this phenomenon remains elusive. In this work...
arXiv:2606. 14965v1 Announce Type: new Abstract: Synthetic instance-dependent label noise (IDN) benchmarks are widely used to evaluate noisy-label learning methods, yet existing approaches typically generate noise through imperfect annotators or classifier raters, leaving the source of ambiguity implicit.
By Shadman Islam, Agustinus Kristiadi, Mostafa Milani
The paper presents Baszta, a Polish multi‑label content‑safety classifier trained by fine‑tuning the 124M‑parameter allegro/herbert‑base‑cased model on five categories (hate, vulgarity, sexual content, crime, self‑harm) using a Focal + R‑Drop objective. In out‑of‑distribution evaluation on the Gadzi Język benchmark, Baszta achieves a small but statistically significant improvement in micro‑F1 over the Bielik Guard system, though the macro‑F1 advantage disappears when both models are properly tuned. The study also explores calibration techniques, showing that per‑category temperature scaling can recover performance lost by Platt scaling or isotonic regression, and discusses the trade‑offs between robust calibration and adversarial recall.
By Adam G\'orski, Mateusz J\k{a}kalak, Rafa{\l} Jakubowski
arXiv:2607. 01715v1 Announce Type: new Abstract: Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level.
By Xudong Wu, Jian Qian, Pangpang Liu, Vaneet Aggarwal, Jiayu Chen
arXiv:2607. 12438v1 Announce Type: new Abstract: Deep networks trained with label noise often learn clean structure before memorizing corrupted labels.
By Satwik Bathula, Anand A. Joshi
The paper investigates how to combine two commonly used classification performance metrics into a single ranking that balances them optimally. It shows that all such metrics can be expressed as ranking scores parameterized by the relative importance of true/false positives and negatives, and derives an exact analytical solution for the optimal tradeoff between any pair of these scores when class priors are equal. The authors also explore experimentally how the solution behaves when performance follows certain distributions with fixed priors.
By S\'ebastien Pi\'erard, Adrien Deli\`ege, Marc Van Droogenbroeck