The paper investigates how repeatedly training score-based diffusion models on synthetic data can cause their outputs to drift from the target distribution. It provides theoretical upper and lower bounds on the divergence accumulated over successive training rounds, distinguishing regimes based on score estimation error and the fraction of fresh data used. Empirical experiments on synthetic data and images support the theoretical findings.
arXiv:2310. 05264v5 Announce Type: replace Abstract: In this work, we investigate an intriguing and prevalent phenomenon of diffusion models which we term as "consistent model reproducibility": given the same starting noise input and a deterministic sampler, different diffusion models often yield remarkably similar outputs.
arXiv:2606. 23920v1 Announce Type: cross Abstract: The task of compositional generation involves using a conditional generative model, trained only on a subset of the possible conditions, to produce samples from compositionally-defined target distributions such as a geometric combination of the source distributions.
The paper introduces a new method for training data attribution in diffusion models called TID, which uses a local score discrepancy measure and can be estimated without retraining. It further distills this approach into TIDE, a forward‑only student that reproduces the teacher’s rankings using internal activations, achieving comparable accuracy at dramatically lower query cost. Experiments on CIFAR‑10, ArtBench‑10, and MS‑COCO show that TID outperforms existing methods and TIDE attributes samples in milliseconds, faster than generation itself.
arXiv:2607. 23226v1 Announce Type: new Abstract: Despite the empirical success of score-based diffusion models, a complete theoretical understanding of how finite-sample learning, network parameterization, and numerical discretization jointly dictate generative quality remains underdeveloped.
arXiv:2608. 18237v1 Announce Type: cross Abstract: Estimating the difference of two Stein's score functions is a fundamental problem in generative modeling.
The paper investigates why diffusion models, despite their practical success, can fully memorize training data when optimal. It shows that as training proceeds, the denoising objective causes overfitting at intermediate noise levels, creating a gap between validation and training performance. In a 2D toy model, this gap arises from the interplay between model error and the data distribution’s support density, leading to a smooth, generalizing flow field rather than exact recall of training points. The study also reveals that predictions from intermediate inference states become increasingly similar to training data in feature space, even as overall similarity to validation data remains unchanged.
arXiv:2610.14136v2 Announce Type: new Abstract: Iterative sampling in diffusion models can lead to error accumulation due to exposure bias. The denoiser is trained on real images interpolated with no...
arXiv:2608. 02575v1 Announce Type: new Abstract: Diffusion models rely on stochastic inputs, yet on finite-precision hardware, the "randomness" they consume is realized as deterministic numerical orbits generated by pseudorandom rules.
The paper investigates memorization in diffusion models, showing that the empirical score function is a weighted sum of Gaussian score functions with sharp softmax weights, causing individual training samples to dominate and lead to sampling collapse. By approximating this function with a neural network, the authors obtain a smoother representation that generalizes better. They introduce two techniques—Noise Unconditioning and Temperature Smoothing—to further reduce single‑sample dominance, and demonstrate improved generalization across multiple datasets while preserving generation quality.
arXiv:2605. 13175v2 Announce Type: replace Abstract: Recent works have proposed incorporating heavy-tailed (HT) noise into diffusion- and flow-based generative models, with the goals of better recovering the tails of target distributions and improving generative diversity.
arXiv:2510. 17917v2 Announce Type: replace-cross Abstract: Data unlearning aims to remove the influence of specific training samples from a trained model.
The paper studies memorization and generalization in conditional diffusion models using a random-feature conditional score model in a high-dimensional setting. It derives asymptotic expressions for training and test losses, showing that in overparameterized regimes wider models better predict the condition-dependent mean but also exhibit reduced within-condition variance, a behavior termed "malign generalization." The analysis also reveals that more informative conditions cause training samples to be memorized at smaller widths, with experimental validation on U‑Net architectures.