arXiv AI
Jun 9
Explaining Data Mixing Scaling Laws
arXiv:2606. 08167v1 Announce Type: cross Abstract: Recent research has established empirical scaling laws to predict model performance on multi-domain data mixtures.
When do data mixtures improve scaling laws? Insights from high-dimensional regression
Related stories
arXiv Machine Learning
Jun 3
Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
arXiv:2603. 05691v3 Announce Type: replace Abstract: It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models.
arXiv Machine Learning
Aug 10
High-dimensional ridgeless least squares interpolation under spiked covariance structures
arXiv:2608. 07281v1 Announce Type: cross Abstract: This paper investigates the asymptotic behavior of the out-of-sample prediction risk of the high-dimensional ridgeless least-squares estimator when the feature dimension $p$ and the sample size $n$ grow proportionally.
arXiv Machine Learning
Sep 10
SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws
The paper analyzes training dynamics of multiclass logistic regression on high‑dimensional Gaussian mixture models with many classes. It finds that learning proceeds sequentially from the most to the least frequent classes and, when class priors follow a power‑law, the cross‑entropy risk evolves through an initial plateau, a power‑law decay phase, and a final convergence phase. The study also shows how model capacity and optimization trade‑off under a fixed compute budget, leading to a compute‑optimal scaling law that prescribes model size and training time as functions of compute.
arXiv AI
Aug 26
Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining
The article proposes treating large language model (LLM) data mixing as a classical mixture experiment, where data domains are components, token shares are proportions, and proxy-training runs serve as design points. Using sparse second‑order Scheffé response‑surface models, the authors construct model‑robust Σ‑optimal designs that efficiently identify optimal data mixtures and reveal strong interaction effects, especially between weak domains and web‑derived text. Empirical results on RegMix show that these designs recover mixture rankings while reducing proxy runs by about 25%, demonstrating that data mixing can be optimized through experimental design rather than solely prediction.
arXiv Machine Learning
Aug 31
Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy
arXiv:2608. 28564v1 Announce Type: cross Abstract: We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent $\alpha\geq 0$ for polynomial inner-product kernels.
arXiv Machine Learning
Jun 10
Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization
arXiv:2509. 17251v2 Announce Type: replace-cross Abstract: Existing theory suggests that for linear regression problems categorized by capacity and source conditions, gradient descent (GD) is always minimax optimal, while both ridge regression and online stochastic gradient descent (SGD) are polynomially suboptimal for certain categories of such problems.
arXiv Machine Learning
Jul 14
Exact Dynamics of Multi-class Stochastic Gradient Descent
arXiv:2510. 14074v2 Announce Type: replace-cross Abstract: We develop a framework for analyzing the learning dynamics of high-dimensional problems trained using one-pass stochastic gradient descent (SGD) with data from multiple anisotropic classes.
arXiv Machine Learning
Jul 20
Prediction-Only Distillation in Linear and Logistic Regression
arXiv:2607. 15450v1 Announce Type: cross Abstract: Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs.
arXiv Machine Learning
Jun 2
Optimal Regularization for Performative Learning
arXiv:2510. 12249v2 Announce Type: replace Abstract: In performative learning, the data distribution reacts to the deployed model - for example, because strategic users adapt their features to game it - which creates a more complex dynamic than in classical supervised learning.
arXiv Machine Learning
Jun 8
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
arXiv:2606. 06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus.
arXiv Machine Learning
Jul 14
Domain-Aware Scaling Laws Uncover Data Synergy
arXiv:2607. 11052v1 Announce Type: new Abstract: Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential.