← Back to all news
arXiv AI By Bao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri

Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data

Read the original on arXiv AI →

The paper investigates when language diffusion models, specifically Uniform-based Discrete Diffusion Models (UDDMs), shift from memorizing training data to generalizing to new data. It shows that UDDMs act as associative memories, forming basins of attraction around stored examples without requiring an explicit energy function. By measuring token recovery and conditional entropy, the authors identify a sharp transition governed by training set size, where memorization (vanishing entropy) gives way to generalization (finite entropy).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 9

Early Signatures of Memorization in Diffusion Models via Basin Geometry and Cyclic Denoising

The paper demonstrates that diffusion models encode memorization in the geometry of their learned energy landscape before it becomes visible in generated samples. By analyzing score divergence and basin volume, the authors show that localized basins form around training images early in training, and that cyclic denoising can recover these images even when one‑shot generation shows no copies. Experiments on Gaussian mixtures, CelebA, CIFAR‑10, and Stable Diffusion v1.4 confirm that basin geometry reveals memorization across various settings.

By Nikhil Verma, Siddharthan Dileep, Anoop Singh, Srikanth Sastry, Ramya Hebbalaguppe, Sayan Ranu, N. M. Anoop Krishnan
diffusion
More like this →
arXiv Machine Learning
Oct 28

Information-Theoretic Scaling Laws for Generative Diffusion Training

The paper introduces an information-theoretic measure called model information to quantify how much a diffusion model’s outputs depend on its specific training dataset. By analyzing how this measure scales with model size and dataset size, the authors identify two distinct regimes: a nonlinear regime where memorization transitions to imperfect generalization, and a linear regime where generalization improves and sample quality rises. They propose a scaling law that captures these regimes and demonstrate that it can predict memorization and sample quality beyond the fitted data range.

By Druv Pai, Dejan Stancevic, Yi Ma, Luca Ambrogioni
diffusion
More like this →
arXiv Machine Learning
Jun 10

The Emergence of Reproducibility and Generalizability in Diffusion Models

arXiv:2310. 05264v5 Announce Type: replace Abstract: In this work, we investigate an intriguing and prevalent phenomenon of diffusion models which we term as "consistent model reproducibility": given the same starting noise input and a deterministic sampler, different diffusion models often yield remarkably similar outputs.

By Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, Qing Qu
diffusionfine-tuningsafety
More like this →