Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. The paper provides end‑to‑end statistical guarantees for transformer‑based MDMs, showing that when the data distribution is well approximated by mixtures of hidden Markov trees, the model can achieve ε‑accurate sampling in KL divergence with ∫O(LK_par/ε) training samples, where K_par is polynomial in the sequence length, vocabulary size, number of hidden states, and mixture components. The transformer network and sampling procedure adapt automatically to the unknown latent structure without requiring prior knowledge of tree topologies or component parameters.
arXiv:2610.07655v1 Announce Type: cross Abstract: Discrete diffusion models have emerged as a practically successful framework for generative modeling on discrete product spaces, yet their statistica...
The paper introduces random walks on graphs as a controllable sandbox for studying parallel sampling strategies in masked diffusion models (MDMs). It shows that common parallel unmasking methods, such as lowest‑entropy scoring, are not universally superior to random sampling and that performance depends on the graph’s conditional dependence structure. The authors also propose training‑free bisection samplers that are provably exact for random walks and demonstrate their effectiveness on both graph‑walk tasks and pretrained MDMs for language generation and reasoning benchmarks.
arXiv:2602. 15008v2 Announce Type: replace Abstract: Diffusion models over discrete spaces have recently shown striking empirical success, yet their theoretical foundations remain incomplete.
The paper compares the parallelism capabilities of three diffusion large language model paradigms—masked, uniform, and Gaussian diffusion. It proves that uniform and Gaussian diffusion can sample with a number of forward passes scaling with the dual total correlation of the distribution, potentially much less than the context length, whereas masked diffusion may require more passes. The study establishes a provable separation in parallelism, showing that masked diffusion’s critical windows are asymptotically narrower than those of the other two approaches.
arXiv:2605. 13026v2 Announce Type: replace-cross Abstract: Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling.
arXiv:2602. 10314v2 Announce Type: replace Abstract: Masked Diffusion Models (MDMs) have emerged as a promising approach for generative modeling in discrete spaces.
The paper presents sample complexity bounds for sampling high‑dimensional categorical distributions with local dependence using discrete diffusions. It introduces a pinning decomposition of the discrete score, enabling a weight‑sharing neural score learner combined with τ‑leaping for efficient end‑to‑end sampling. The authors derive explicit guarantees that depend on vocabulary size, MRF interaction order, and sample size, and demonstrate that weight‑sharing networks outperform fully connected ones on Potts, Ising, and tree‑structured models.
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
arXiv:2607. 27529v1 Announce Type: new Abstract: Discrete diffusion and flow-matching models denoise a sequence over many steps, but to keep each step cheap, they factorize the transition across positions and decide every token independently.
arXiv:2605. 17232v5 Announce Type: replace Abstract: Discrete diffusion has become a leading framework for generative modeling in various applications including language, vision, and biology.
arXiv:2606. 02241v1 Announce Type: new Abstract: Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion?
arXiv:2609.37974v1 Announce Type: cross Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The m...