arXiv Machine Learning September 17, 2026 By Matteo Marchi, Jo\~ao Pedro Silvestre, Bahman Gharesifard, Paulo Tabuada

Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data

Read the original on arXiv Machine Learning →

The paper investigates how to avoid model collapse when training large language models with synthetic data. It establishes theoretical guarantees for the minimum ratio of human to synthetic data needed to maintain training stability, using the Fisher‑Rao metric to analyze dynamics on the probability simplex. The authors derive contraction and invariance bounds that remain meaningful even in high dimensions, showing that the required data ratio differs from earlier estimates.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jun 17

Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework

arXiv:2502. 18049v5 Announce Type: replace-cross Abstract: Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation.

By Hengzhi He, Shirong Xu, Guang Cheng

arXiv Machine Learning

Jul 17

Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

arXiv:2510. 16657v3 Announce Type: replace-cross Abstract: Synthetic data has been increasingly used to train frontier generative models.

By Bingji Yi, Qiyuan Liu, Yuwei Cheng, Haifeng Xu

arXiv Statistics ML

Sep 10

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

arXiv:2609.09572v1 Announce Type: new Abstract: Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Doh...

By Jichu li, Difan Zou

safety

arXiv Machine Learning

Jul 7

Language Generation with Replay: A Learning-Theoretic View of Model Collapse

arXiv:2603. 11784v2 Announce Type: replace Abstract: As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed.

By Giorgio Racca, Michal Valko, Amartya Sanyal

llms

arXiv AI

Jun 30

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data

arXiv:2606. 30372v1 Announce Type: new Abstract: Quantitative research across the social and behavioral sciences depends on human subject experiments that are expensive, slow, and subject to sampling bias.

By Haobo Yang

llms safety

arXiv Machine Learning

Jul 20

Retraining Seeks Stable Signals

arXiv:2607. 15623v1 Announce Type: cross Abstract: Predictive models deployed at scale influence future data, a phenomenon called performativity.

By Moritz Hardt

llms

One email a morning, machine-written

Related stories

Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework

Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

Language Generation with Replay: A Learning-Theoretic View of Model Collapse

Using Large Language Models as Low-Cost Statistical Estimators for Human-Response Data

Retraining Seeks Stable Signals