← Back to all news
arXiv Machine Learning By Jingda Wu, Changxiao Cai

Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure

Read the original on arXiv Machine Learning →

Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. The paper provides end‑to‑end statistical guarantees for transformer‑based MDMs, showing that when the data distribution is well approximated by mixtures of hidden Markov trees, the model can achieve ε‑accurate sampling in KL divergence with ∫O(LK_par/ε) training samples, where K_par is polynomial in the sequence length, vocabulary size, number of hidden states, and mixture components. The transformer network and sampling procedure adapt automatically to the unknown latent structure without requiring prior knowledge of tree topologies or component parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
6d ago

Walk fast but be careful: Understanding Parallel Sampling in Masked Diffusion

The paper introduces random walks on graphs as a controllable sandbox for studying parallel sampling strategies in masked diffusion models (MDMs). It shows that common parallel unmasking methods, such as lowest‑entropy scoring, are not universally superior to random sampling and that performance depends on the graph’s conditional dependence structure. The authors also propose training‑free bisection samplers that are provably exact for random walks and demonstrate their effectiveness on both graph‑walk tasks and pretrained MDMs for language generation and reasoning benchmarks.

By Vansh Bansal, Cholyeon Cho, Syamantak Kumar, Sujay Sanghavi, Purnamrita Sarkar
diffusionbenchmarks
More like this →
arXiv Machine Learning
Sep 18

Parallelism, critical windows, and separations among diffusion language models

The paper compares the parallelism capabilities of three diffusion large language model paradigms—masked, uniform, and Gaussian diffusion. It proves that uniform and Gaussian diffusion can sample with a number of forward passes scaling with the dual total correlation of the distribution, potentially much less than the context length, whereas masked diffusion may require more passes. The study establishes a provable separation in parallelism, showing that masked diffusion’s critical windows are asymptotically narrower than those of the other two approaches.

By Sitan Chen, Liye Wang
llmsdiffusion
More like this →