arXiv Statistics ML
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Jun 17
Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
arXiv:2502. 18049v5 Announce Type: replace-cross Abstract: Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation.
arXiv AI
2d ago
Training-Aware Target Coverage for Synthetic Data Selection
arXiv:2610.00814v1 Announce Type: cross Abstract: Synthetic data are increasingly used to scale LLM training, yet more synthetic data do not necessarily produce better models. Useful synthetic data m...
arXiv Machine Learning
Jul 17
Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
arXiv:2510. 16657v3 Announce Type: replace-cross Abstract: Synthetic data has been increasingly used to train frontier generative models.
arXiv Machine Learning
Jul 16
When T2I Synthetic Data Backfires: Amplified Privacy Risks in Real-Synthetic Mix Training
arXiv:2607. 13541v1 Announce Type: cross Abstract: To overcome data scarcity and privacy constraints in data collection, it has become standard practice across academia and industry to augment real training data with text-to-image (T2I)-generated synthetic data, a paradigm we term Real-Synthetic Mix-Training (RSMT).
arXiv Machine Learning
Aug 11
Coupled Training with Privileged Information and Unlabeled Data
arXiv:2605. 23268v2 Announce Type: replace-cross Abstract: In many prediction problems, we have extra information during training (for example, measurements that are expensive or slow to collect) that will not be available when the model is deployed.
arXiv AI
Sep 10
Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
The paper introduces SynPro, a synthetic data generation framework that augments limited organic text for large language model pretraining by applying rephrasing and reformatting operations. SynPro’s generators are optimized with reinforcement learning rewards for quality, faithfulness, and data influence, and are updated continuously as training plateaus. Experiments on 400M, 1.1B, and 2B models show that SynPro can unlock 3.4–5.2× the effective tokens of standard repetition, even outperforming a non‑data‑bound oracle at larger scales.