The paper presents a high‑dimensional theory of learning with data pruning via weighted empirical risk minimization. It analyzes pruning along a one‑dimensional subspace and derives limiting test‑error curves as input dimension and sample size grow with a fixed ratio. The authors show that, in the high‑sample regime, the optimal strategy is to keep only the 2N hardest examples, achieving a Bayes‑optimal error scaling of order N/P instead of the usual sqrt(N/P).
Despite a surge of empirical interest in the ``unreasonable effectiveness'' of drastic data pruning for AI systems, a unified theoretical understanding of this phenomenon remains elusive. In this work...
arXiv:2412.00143v3 Announce Type: replace-cross Abstract: Oracle pruning, which selects unimportant weights by minimizing the pruned train loss, has served as the foundation for most neural network p...
arXiv:2606. 11761v1 Announce Type: new Abstract: Dynamic data pruning techniques aim to reduce computational cost while minimizing information loss by periodically selecting representative subsets of input data during model training.
arXiv:2606. 08574v1 Announce Type: new Abstract: Data pruning (DP), as an oft-stated strategy to alleviate heavy training burdens, reduces the volume of training samples according to a well-defined pruning method while striving for near-lossless performance.
arXiv:2609.10311v1 Announce Type: cross Abstract: The lottery ticket hypothesis posits the existence of winning tickets: sparse subnetworks that, when trained in isolation from their original initial...
arXiv:2606. 07615v1 Announce Type: cross Abstract: Deep neural networks often contain redundant hidden units.
arXiv:2602. 11557v2 Announce Type: replace Abstract: A variety of widely used optimization methods like SignSGD and Muon can be interpreted as instances of steepest descent under different norm-induced geometries.
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
arXiv:2603. 06957v2 Announce Type: replace-cross Abstract: We study post-training linear autoregressive models with outcome and process rewards.
arXiv:2606. 11149v1 Announce Type: new Abstract: We study the problem of learning a drifting concept in the presence of Massart noise.
arXiv:2603. 02234v3 Announce Type: replace-cross Abstract: The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is sufficient.
The paper introduces Rank‑Variance Scores (TRV) as a new metric for importance sampling and data pruning in the presence of label contamination. It shows that traditional loss‑based methods fail because corrupted examples often have high loss, and proves that a useful score must be non‑monotonic in loss. By ranking examples across multiple independently trained proxy models, TRV captures both between‑proxy disagreement and within‑proxy variance, enabling Disagreement‑Regularized Importance Sampling (DR‑IS) and TRV‑based pruning to outperform random selection on noisy CIFAR‑10/100 datasets.