The paper investigates why large neural networks outperform smaller ones, focusing on a phenomenon called primacy bias—where early data distributions hinder learning of later ones. It shows that overparameterized models are more robust to this bias, especially during sequential pretraining on heterogeneous data. The authors propose Exposure Therapy, a regularization technique that improves small models’ performance on late data and overall capabilities up to the billion‑parameter scale.
By Mohnish Harwani, Yujia Zheng
The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.
By Jiazheng Li, Huaqing Zhang, Jing Xu, Tianle Cai, Jingzhao Zhang
The paper titled "Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases" examines how training on a smaller dataset with repeated samples can reduce computational cost compared to using a larger dataset. Across various tasks, architectures, and optimizers, this phenomenon cannot be explained by existing theory. The authors attribute the speedup to layer‑wise growth driven by sampling biases, which is more pronounced with smaller datasets, and provide both theoretical analysis and empirical evidence to support this claim.
By Jingwen Liu, Ezra Edelman, Surbhi Goel, Bingbin Liu
arXiv:2606. 07527v1 Announce Type: cross Abstract: The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL.
By Michael Hassid, Yossi Adi, Roy Schwartz
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.
By Jing Huang, Daniel Wurgaft, Rachit Bansal, Laura Ruis, Naomi Saphra, David Alvarez-Melis, Andrew Kyle Lampinen, Christopher Potts, Ekdeep Singh Lubana
The paper demonstrates that a tabular foundation model can achieve strong generalization using only a single real table for self‑supervised pre‑training, challenging the belief that large synthetic or real datasets are necessary. By systematically pre‑training and evaluating across diverse benchmarks, the authors show that the number and quality of tasks that can be derived from a dataset are critical for downstream performance. This finding suggests that carefully constructed task sets from limited data can enable effective transfer learning in tabular models.
By Junwei Ma, Nour Shaheen, Alex Labach, Amine Mhedhbi, Frank Hutter, Anthony L. Caterini, Valentin Thomas
Pre-training has become a fundamental paradigm in modern machine learning, with one of its key empirical benefits being reduced downstream sample complexity as the scale of pre-training data increases. However, existing theoretical frameworks for pre-training do not fully explain this phenomenon.
The paper introduces SynPro, a synthetic data generation framework that augments limited organic text for large language model pretraining by applying rephrasing and reformatting operations. SynPro’s generators are optimized with reinforcement learning rewards for quality, faithfulness, and data influence, and are updated continuously as training plateaus. Experiments on 400M, 1.1B, and 2B models show that SynPro can unlock 3.4–5.2× the effective tokens of standard repetition, even outperforming a non‑data‑bound oracle at larger scales.
By Zichun Yu, Chenyan Xiong
arXiv:2606. 02008v1 Announce Type: cross Abstract: Pre-training has become a fundamental paradigm in modern machine learning, with one of its key empirical benefits being reduced downstream sample complexity as the scale of pre-training data increases.
By Kazuto Fukuchi, Ryuichiro Hataya, Kota Matsui
arXiv:2510. 01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.
By Wa\"iss Azizian, Ali Hasan
arXiv:2502.06970v3 Announce Type: replace
Abstract: In contemporary deep learning, a prevalent and effective workflow for solving low-data problems is adapting powerful pre-trained foundation models...
By Fady Rezk, Royson Lee, Henry Gouk, Timothy Hospedales, Minyoung Kim
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization.