arXiv Machine Learning

Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models

arXiv AI
Oct 15

Correction Training for Diffusion Models

arXiv:2610.14136v2 Announce Type: new Abstract: Iterative sampling in diffusion models can lead to error accumulation due to exposure bias. The denoiser is trained on real images interpolated with no...

By Donghoon Ahn, Runqian Wang, Alexei A. Efros, Taesung Park, Yash Belhe
arXiv Machine Learning
Sep 10

Noise in Diffusion Models Is a Learnable Input

The paper argues that the concrete random noise used in diffusion models is not merely a passive perturbation but a learnable input that can be exploited by the model. By analyzing how clean data and realized noise jointly form the noisy input, the authors show that the model can learn regularities in the data or in the noise structure, and that these two routes can interact. Experiments on MNIST and CIFAR‑10 using pseudorandom streams demonstrate that structured‑noise training can reduce prediction loss, but this advantage disappears when test noise is replaced with IID noise, indicating that the learned dependence is tied to the specific noise structure.

By Shengzhi Deng, Chenqi Ye, Yanze Guo
arXiv AI
Oct 19

Rethinking the design space of Vanilla Pixel Diffusion Transformers

The paper revisits pixel-space diffusion models, showing that a vanilla diffusion transformer can generate directly in pixel space by swapping velocity prediction for clean image prediction. It introduces MFlow, an adaptive parameterization that interpolates between these two prediction modes based on data representation and model capacity, and demonstrates competitive performance on ImageNet, UCF‑101, and text‑to‑image tasks. Additionally, the authors add ConvRefiner, a lightweight convolutional module trained post‑hoc to restore high‑frequency details while preserving FID.

By Soumik Mukhopadhyay, Long Mai, Abhinav Shrivastava, Aniruddha Mahapatra
arXiv Machine Learning
Oct 9

Beyond Sample Copying: Structural Memorization in Diffusion Models

The paper investigates why diffusion models, despite their practical success, can fully memorize training data when optimal. It shows that as training proceeds, the denoising objective causes overfitting at intermediate noise levels, creating a gap between validation and training performance. In a 2D toy model, this gap arises from the interplay between model error and the data distribution’s support density, leading to a smooth, generalizing flow field rather than exact recall of training points. The study also reveals that predictions from intermediate inference states become increasingly similar to training data in feature space, even as overall similarity to validation data remains unchanged.

By Tim Kaiser, Markus Kollmann
arXiv Machine Learning
Oct 8

Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers

The paper investigates why Diffusion Transformers perform well when predicting clean data but poorly when predicting noise or velocity. It introduces the concept of *residual-stream burden*, explaining that noisy targets force the residual stream to carry noise-dependent variation through the network, limiting representation flexibility. Experiments show that spectral concentration of clean targets reduces this burden, and the authors propose Spatially Indexed Hyper-Connections (SiHC) to expand residual bandwidth, achieving state‑of‑the‑art FID on ImageNet.

By Tongtong Liang, Siqi Kou, Ziqiao Xi, Esha Singh, Kun Zhou, Zhijie Deng, Alexander Cloninger, Yu-Xiang Wang, Rahul Parhi
arXiv Machine Learning
Nov 5

Quantifying Error Propagation and Model Collapse in Diffusion Models

The paper investigates how repeatedly training score-based diffusion models on synthetic data can cause their outputs to drift from the target distribution. It provides theoretical upper and lower bounds on the divergence accumulated over successive training rounds, distinguishing regimes based on score estimation error and the fraction of fresh data used. Empirical experiments on synthetic data and images support the theoretical findings.

By Nail B. Khelifa, Richard E. Turner, Ramji Venkataramanan
arXiv Machine Learning
Jun 10

The Emergence of Reproducibility and Generalizability in Diffusion Models

arXiv:2310. 05264v5 Announce Type: replace Abstract: In this work, we investigate an intriguing and prevalent phenomenon of diffusion models which we term as "consistent model reproducibility": given the same starting noise input and a deterministic sampler, different diffusion models often yield remarkably similar outputs.

By Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, Qing Qu