The paper investigates why diffusion models sometimes fail to learn symmetries present in data distributions, such as different orientations of a dog. Focusing on linear diffusion models, it identifies three possible sources of symmetry breaking—non‑symmetric initial weights, noisy learning targets, and finite‑sample noise—and shows that target noise is the most robust cause, persisting even with abundant data. Using random matrix theory, the authors derive formulas for average symmetry error, revealing that smaller symmetries are harder to learn.
arXiv:2602. 02908v2 Announce Type: replace-cross Abstract: Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed.
arXiv:2605.06140v3 Announce Type: replace-cross Abstract: Generative modeling of physical systems, such as molecules, requires learning distributions that are invariant under global symmetries, such...
The paper introduces an information-theoretic measure called model information to quantify how much a diffusion model’s outputs depend on its specific training dataset. By analyzing how this measure scales with model size and dataset size, the authors identify two distinct regimes: a nonlinear regime where memorization transitions to imperfect generalization, and a linear regime where generalization improves and sample quality rises. They propose a scaling law that captures these regimes and demonstrate that it can predict memorization and sample quality beyond the fitted data range.
The paper investigates how limited model capacity forces symmetry breaking in problems that are inherently rotation‑invariant. It shows that when a model’s capacity cuts across a symmetry block, the optimal solution must abandon symmetry, and quantifies the exact loss incurred for a Gaussian bilinear predictor. Experiments on attention heads confirm that the predicted symmetry‑breaking behavior occurs at the expected head dimensions and rotation sensitivities.
arXiv:2510. 17303v2 Announce Type: replace Abstract: Symmetries are known to improve the empirical performance of machine learning models, yet theoretical guarantees explaining these gains remain limited.
arXiv:2505. 19619v3 Announce Type: replace Abstract: Deep generative models have recently garnered significant attention across various fields, from physics to chemistry, where sampling from unnormalized Boltzmann-like distributions represents a fundamental challenge.
arXiv:2603. 12901v2 Announce Type: replace-cross Abstract: While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood.
arXiv:2310. 05264v5 Announce Type: replace Abstract: In this work, we investigate an intriguing and prevalent phenomenon of diffusion models which we term as "consistent model reproducibility": given the same starting noise input and a deterministic sampler, different diffusion models often yield remarkably similar outputs.
arXiv:2608. 01582v1 Announce Type: cross Abstract: Symmetry is central to modern machine learning and physics: invariances and equivariances improve sample efficiency, robustness, and out-of-distribution generalization, while symmetry principles guide scientific modeling.
arXiv:2512. 20043v3 Announce Type: replace Abstract: Symmetry is fundamental to understanding physical systems and can improve performance and sample efficiency in machine learning.
arXiv:2608. 13457v1 Announce Type: new Abstract: Generating crystals has recently attracted significant interest due to their broad applications in materials science.
arXiv:2605. 13175v2 Announce Type: replace Abstract: Recent works have proposed incorporating heavy-tailed (HT) noise into diffusion- and flow-based generative models, with the goals of better recovering the tails of target distributions and improving generative diversity.