arXiv Machine Learning

Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework

arXiv:2502. 18049v5 Announce Type: replace-cross Abstract: Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation.

arXiv Machine Learning

Sep 17

Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data

The paper investigates how to avoid model collapse when training large language models with synthetic data. It establishes theoretical guarantees for the minimum ratio of human to synthetic data needed to maintain training stability, using the Fisher‑Rao metric to analyze dynamics on the probability simplex. The authors derive contraction and invariance bounds that remain meaningful even in high dimensions, showing that the required data ratio differs from earlier estimates.

By Matteo Marchi, Jo\~ao Pedro Silvestre, Bahman Gharesifard, Paulo Tabuada

llms

arXiv Machine Learning

Jul 17

Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

arXiv:2510. 16657v3 Announce Type: replace-cross Abstract: Synthetic data has been increasingly used to train frontier generative models.

By Bingji Yi, Qiyuan Liu, Yuwei Cheng, Haifeng Xu

arXiv Statistics ML

Sep 10

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

arXiv:2609.09572v1 Announce Type: new Abstract: Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Doh...

By Jichu li, Difan Zou

safety

arXiv AI

2d ago

Rethinking Data Augmentation under Covariate Shift: Invariant-Guided Diffusion and Prototype Reweighting

arXiv:2610.00873v1 Announce Type: cross Abstract: In many industrial applications, 1) tabular data is scarce and imbalanced and thus requires synthetic expansion; 2) input distributions drift between...

By Hongyu Cao, Xinyuan Wang, Arun Vignesh Malarkkan, Kunpeng Liu, Haifeng Chen, Yanjie Fu

diffusion data-engineering

arXiv Machine Learning

Jun 15

Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization

arXiv:2606. 13796v1 Announce Type: cross Abstract: Recursive training of generative models on their own outputs can lead to model collapse, a compounding drift away from the true data distribution.

By Na\"il B. Khelifa, Richard E. Turner, Ramji Venkataramanan

diffusion

arXiv AI

Jun 16

Quantifying the Impact of Lossy Compression on Neural Generative Surrogate Modeling

arXiv:2606. 15959v1 Announce Type: cross Abstract: Neural networks are used as generative surrogate models for scientific discovery, which are trainable approximations of scientific simulations.

By Zhimin Li, Harshitha Menon, Charles Jekel, Valerio Pascucci, Peter Lindstrom

arXiv Machine Learning

Jun 16

Discriminative Span as a Predictor of Synthetic Data Utility via Classifier Reconstruction

arXiv:2605. 09697v3 Announce Type: replace-cross Abstract: In many real-world computer vision applications, including medical imaging and industrial inspection, binary classification tasks are characterized by a severe scarcity of positive samples.

By Radhika Amar Desai, Modigari Narendra

rag computer-vision data-engineering

arXiv AI

Jun 2

Efficient Weighted Sampling via Score-based Generative Models

arXiv:2502. 04646v2 Announce Type: replace-cross Abstract: Weighted sampling -- sampling from a probability density function (PDF) proportional to the product of a base PDF and a weight function -- is a fundamental technique with wide-ranging applications in variance reduction, biased sampling, data augmentation, and more.

By Heasung Kim, Taekyun Lee, Hyeji Kim, Gustavo de Veciana

diffusion benchmarks

arXiv Machine Learning

Aug 11

From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Spurious Correlations in Image Recognition

arXiv:2308. 04553v4 Announce Type: replace-cross Abstract: Visual recognition models are prone to learning spurious correlations induced by a biased training set where certain conditions $B$ (\eg, Indoors) are over-represented in certain classes $Y$ (\eg, Big Dogs).

By Maan Qraitem, Kate Saenko, Bryan A. Plummer

computer-vision benchmarks safety

arXiv Machine Learning

Jun 9

ReTabSyn: Realistic Tabular Data Synthesis via Reinforcement Learning

arXiv:2603. 10823v2 Announce Type: replace-cross Abstract: Deep generative models can help with data scarcity and privacy by producing synthetic training data, but they struggle in low-data, imbalanced tabular settings to fully learn the complex data distribution.

By Xiaofeng Lin, Seungbae Kim, Zhuoya Li, Zachary DeSoto, Charles Fleming, Guang Cheng

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jul 7

Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models

arXiv:2607. 04360v1 Announce Type: cross Abstract: Conditional generative models have emerged as powerful tools for sampling from target conditional distributions, driving substantial advances across a wide range of scientific and applied domains.

By Shijin Gong, Baihua He, Xinyu Zhang

arXiv Machine Learning

Jul 2

Decision-Aware Training for Sample-Based Generative Models

arXiv:2607. 01171v1 Announce Type: new Abstract: Sample-based generative models are increasingly used for probabilistic forecasting in high-stakes decision settings, yet their training objectives are blind to the decision maker's cost structure.

By Kornelius Raeth, Nicole Ludwig

Related stories