Hugging Face Trending Papers
Sequential Pretraining Favors Large Models
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Oct 8
Sequential Pretraining Favors Large Models
The paper investigates why large neural networks outperform smaller ones, focusing on a phenomenon called primacy bias—where early data distributions hinder learning of later ones. It shows that overparameterized models are more robust to this bias, especially during sequential pretraining on heterogeneous data. The authors propose Exposure Therapy, a regularization technique that improves small models’ performance on late data and overall capabilities up to the billion‑parameter scale.
arXiv AI
Oct 26
Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting
The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.
arXiv AI
Sep 7
Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
The paper titled "Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases" examines how training on a smaller dataset with repeated samples can reduce computational cost compared to using a larger dataset. Across various tasks, architectures, and optimizers, this phenomenon cannot be explained by existing theory. The authors attribute the speedup to layer‑wise growth driven by sampling biases, which is more pronounced with smaller datasets, and provide both theoretical analysis and empirical evidence to support this claim.
arXiv AI
Jun 9
Post-training is (Massive) Supervised Learning
arXiv:2606. 07527v1 Announce Type: cross Abstract: The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL.
arXiv Machine Learning
Jun 2
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.
arXiv AI
Sep 15
Generalization Can Emerge in Tabular Foundation Models From a Single Table
The paper demonstrates that a tabular foundation model can achieve strong generalization using only a single real table for self‑supervised pre‑training, challenging the belief that large synthetic or real datasets are necessary. By systematically pre‑training and evaluating across diverse benchmarks, the authors show that the number and quality of tasks that can be derived from a dataset are critical for downstream performance. This finding suggests that carefully constructed task sets from limited data can enable effective transfer learning in tabular models.