arXiv Machine Learning July 31, 2026 By Joel Niklaus, Atsuki Yamaguchi, Michal \v{S}tef\'anik, Guilherme Penedo, Hynek Kydl\'i\v{c}ek, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Read the original on arXiv Machine Learning →

arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 14

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.

By Ajay Patel, Colin Raffel, Chris Callison-Burch

llms benchmarks

arXiv AI

Aug 11

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

arXiv:2603. 22213v2 Announce Type: replace-cross Abstract: While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection.

By Kexian Tang, Jiani Wang, Shaowen Wang, Kaifeng Lyu

llms reinforcement-learning

arXiv Machine Learning

Jun 10

CodeAlchemy: Synthetic Code Rewriting at Scale

arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.

By Ankit Gupta, Aditya Prasad, Rameswar Panda

llms benchmarks

arXiv AI

Jun 11

A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data

arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.

By Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Fariha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, Na Zou

llms

arXiv AI

Jul 10

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

arXiv:2607. 08646v1 Announce Type: cross Abstract: As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish.

By Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai, Jie Zhou, Qiang Ma, Xuanhe Zhou, Xu Han, Yudong Wang, Zhiyuan Liu

llms safety

arXiv Machine Learning

Jun 9

ReTabSyn: Realistic Tabular Data Synthesis via Reinforcement Learning

arXiv:2603. 10823v2 Announce Type: replace-cross Abstract: Deep generative models can help with data scarcity and privacy by producing synthetic training data, but they struggle in low-data, imbalanced tabular settings to fully learn the complex data distribution.

By Xiaofeng Lin, Seungbae Kim, Zhuoya Li, Zachary DeSoto, Charles Fleming, Guang Cheng

llms reinforcement-learning fine-tuning benchmarks safety

Related stories

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

CodeAlchemy: Synthetic Code Rewriting at Scale

A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

ReTabSyn: Realistic Tabular Data Synthesis via Reinforcement Learning