arXiv Machine Learning July 31, 2026 By Joel Niklaus, Atsuki Yamaguchi, Michal \v{S}tef\'anik, Guilherme Penedo, Hynek Kydl\'i\v{c}ek, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Read the original on arXiv Machine Learning →

arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 14

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.

By Ajay Patel, Colin Raffel, Chris Callison-Burch

llms benchmarks

arXiv Machine Learning

Sep 10

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

RePro is a web‑recycling technique that trains a small language model (as little as 1 B parameters) with reinforcement learning to produce high‑quality, faithful rephrasings of pretraining data. The method uses one quality reward and three faithfulness rewards to preserve core semantics and structure while converting organic data into better training examples. Experiments show that RePro boosts downstream accuracy by 3.7–14.5 % over organic‑only baselines and improves data efficiency 2–3×, outperforming prior prompting‑based recycling approaches.

By Zichun Yu, Chenyan Xiong

llms reinforcement-learning benchmarks

arXiv Machine Learning

4d ago

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...

By Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko

llms benchmarks

arXiv AI

Aug 11

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

arXiv:2603. 22213v2 Announce Type: replace-cross Abstract: While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection.

By Kexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu

llms reinforcement-learning

arXiv AI

4d ago

Is Human-Readable Text Necessary for Effective LLM Fine-Tuning?

arXiv:2609.35868v1 Announce Type: new Abstract: Is human readability necessary for effective fine-tuning of large language models? We investigate whether model-conditioned training representations ca...

By Jinhao Zhang, Zeyu Liu, Zicheng Yan, Yunquan Zhang, Daning Cheng, Song Tang

llms rag fine-tuning benchmarks

arXiv AI

Sep 16

A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance

arXiv:2609.16592v1 Announce Type: new Abstract: This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with...

By Kimberly Le Truong, Nari Johnson, Anna Kawakami, Hoda Heidari

llms benchmarks data-engineering

One email a morning, machine-written

Related stories

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

Is Human-Readable Text Necessary for Effective LLM Fine-Tuning?

A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance