← Back to all news
arXiv Machine Learning By Mohnish Harwani, Yujia Zheng

Sequential Pretraining Favors Large Models

Read the original on arXiv Machine Learning →

The paper investigates why large neural networks outperform smaller ones, focusing on a phenomenon called primacy bias—where early data distributions hinder learning of later ones. It shows that overparameterized models are more robust to this bias, especially during sequential pretraining on heterogeneous data. The authors propose Exposure Therapy, a regularization technique that improves small models’ performance on late data and overall capabilities up to the billion‑parameter scale.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 26

Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting

The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.

By Jiazheng Li, Huaqing Zhang, Jing Xu, Tianle Cai, Jingzhao Zhang
llmsbenchmarks
More like this →
arXiv AI
Sep 15

Generalization Can Emerge in Tabular Foundation Models From a Single Table

The paper demonstrates that a tabular foundation model can achieve strong generalization using only a single real table for self‑supervised pre‑training, challenging the belief that large synthetic or real datasets are necessary. By systematically pre‑training and evaluating across diverse benchmarks, the authors show that the number and quality of tasks that can be derived from a dataset are critical for downstream performance. This finding suggests that carefully constructed task sets from limited data can enable effective transfer learning in tabular models.

By Junwei Ma, Nour Shaheen, Alex Labach, Amine Mhedhbi, Frank Hutter, Anthony L. Caterini, Valentin Thomas
llmsbenchmarks
More like this →