Cosmopedia: how to create large-scale synthetic data for pre-training Large Language Models

Databricks ❤️ Hugging Face: up to 40% faster training and tuning of Large Language Models

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.

Very Large Language Models and How to Evaluate Them

Learning Spatio-Temporal Foundation Models from Pure Synthetic Data

arXiv:2607. 16251v1 Announce Type: new Abstract: Spatio-Temporal Foundation Models (STFMs) aim to learn generalizable representations of complex dynamical systems across space and time.

Efficient training of language models to fill in the middle

Towards Engineering Scaling Laws with Pretraining Data Composition

arXiv:2606. 19781v1 Announce Type: cross Abstract: Neural scaling laws describe how model performance improves as a power law in compute, model size, and dataset size.