The paper investigates how benchmark contamination—leakage of test items into training data—affects large language model (LLM) leaderboards. By comparing original test items with semantically equivalent paraphrases, the authors measure contamination as a violation of anchor-item invariance and find that it inflates absolute scores but rarely changes model rankings. Across 47 public models and 74 finetuned models on four benchmarks, the rank correlation between standard and paraphrase-controlled leaderboards is 0.997, with only a handful of cases showing differential contamination that could alter rankings.
By Xingyao Xiao (Stanford University), Yihong Cheng (City University of Macau)
arXiv:2610.14015v1 Announce Type: cross
Abstract: Learning-to-rank (LTR) models can align rankings with undesignated signals when those signals appear as features or leak into training labels. We com...
By Vladyslav Shevchuk, Carlos Uziel P\'erez Malla, Philipp Wissgott
The paper presents a high‑dimensional theory of learning with data pruning via weighted empirical risk minimization. It analyzes pruning along a one‑dimensional subspace and derives limiting test‑error curves as input dimension and sample size grow with a fixed ratio. The authors show that, in the high‑sample regime, the optimal strategy is to keep only the 2N hardest examples, achieving a Bayes‑optimal error scaling of order N/P instead of the usual sqrt(N/P).
By Elvis Dohmatob
arXiv:2610.01028v1 Announce Type: cross
Abstract: Machine learning models often suffer performance degradation under subpopulation shift, particularly when spurious correlations cause models to rely...
By Sung Ho Jo, Seonghwi Kim, Wonsang Yun, Minwoo Chae
We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists.
arXiv:2607. 24622v1 Announce Type: cross Abstract: We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones.
By Gabriel Singer, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis, Argyris Kalogeratos