arXiv AI
Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 27
Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
arXiv:2608. 25826v1 Announce Type: cross Abstract: A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leaves the structure of whole documents untouched.
Hugging Face Trending Papers
Aug 26
Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leav...
arXiv Computation and Language
Aug 25
ConvergeWriter: Data-Driven Bottom-Up Article Construction
ConvergeWriter introduces a bottom‑up, data‑driven framework for long‑form document generation that first retrieves exhaustive knowledge from a source corpus and clusters it into distinct knowledge groups. These clusters then guide the creation of a hierarchical outline and the final text, ensuring the output is strictly grounded in the retrieved material and traceable to its sources. Experiments on 14B and 32B LLMs show that this approach matches or surpasses state‑of‑the‑art baselines, especially in scenarios requiring high factual fidelity and structural coherence.
arXiv Machine Learning
Jul 31
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.
arXiv Machine Learning
Jul 14
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.
arXiv Computation and Language
5d ago
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
AFCS (Atomic‑Fact‑Conditioned Synthesis) is a new pipeline for generating synthetic data for language‑model pretraining that focuses on extracting and using fine‑grained atomic facts from source documents. Unlike previous methods such as WRAP and Nemotron‑CC, AFCS explicitly requires secondary entities, list/table entries, and precise temporal attributes to appear in the synthesized corpus, employing structure‑aware extraction, source‑grounded verification, local context expansion, and multi‑format generation. In experiments with OLMo‑3‑7B and a 32B model on the FineWiki corpus, AFCS outperforms the baselines on several QA benchmarks and improves general benchmark performance by +1.09 when integrated mid‑training with 100B tokens.