The paper introduces the Single-Increment Rejection Sampler (SIRS) to improve query efficiency in parallel masked diffusion models (MDMs). SIRS estimates the log acceptance probability using only one randomly sampled increment, then applies a Bernoulli factory to decide accept/reject outcomes. By Rao‑Blackwellizing each increment and bounding the KL‑divergence, SIRS reduces bias while maintaining parallel decoding performance.
arXiv:2606. 10829v1 Announce Type: cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled.
arXiv:2608.25505v2 Announce Type: replace-cross Abstract: Masked diffusion models generate sequences in rounds, predicting several masked positions at once. Fewer rounds can accelerate decoding, but...
The paper introduces random walks on graphs as a controllable sandbox for studying parallel sampling strategies in masked diffusion models (MDMs). It shows that common parallel unmasking methods, such as lowest‑entropy scoring, are not universally superior to random sampling and that performance depends on the graph’s conditional dependence structure. The authors also propose training‑free bisection samplers that are provably exact for random walks and demonstrate their effectiveness on both graph‑walk tasks and pretrained MDMs for language generation and reasoning benchmarks.
The paper introduces Fork-dLLM, a hybrid sampling strategy for diffusion language models that combines parallel token generation with autoregressive ordering only at uncertain fallback steps, thereby avoiding the flexibility trap that reduces generation diversity. It also presents ForkGRPO, an extension that applies the GRPO objective selectively during post‑training rollouts, preserving policy‑likelihood ratios while cutting rollout and optimization costs. Experiments show Fork-dLLM achieves pass@k scaling comparable to autoregressive sampling but 2–3× more efficient, and ForkGRPO matches or surpasses AR‑based GRPO baselines at significantly lower training expense.
arXiv:2607. 16207v1 Announce Type: new Abstract: Membership inference attacks (MIAs) test whether a candidate example appeared in a model's training data.
Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. The paper provides end‑to‑end statistical guarantees for transformer‑based MDMs, showing that when the data distribution is well approximated by mixtures of hidden Markov trees, the model can achieve ε‑accurate sampling in KL divergence with ∫O(LK_par/ε) training samples, where K_par is polynomial in the sequence length, vocabulary size, number of hidden states, and mixture components. The transformer network and sampling procedure adapt automatically to the unknown latent structure without requiring prior knowledge of tree topologies or component parameters.
The paper introduces a new framework for adaptive parallel sampling of discrete vectors, where a deterministic policy reveals coordinates round‑by‑round based on previously observed values and samples the remaining coordinates from their exact conditional marginals. The authors prove an exact identity linking the forward Kullback‑Leibler divergence of any policy to the expected conditional total correlation accumulated during the sampling process, establishing conditional total correlation as the precise information cost of within‑round parallelism. Using this identity, they derive zero‑error schedules for finite‑order Markov chains, characterize the serial depth of Bernoulli walks, and demonstrate separations between different reveal orders, permutation strategies, and string structures, thereby revealing how conditional dependence governs parallelizability. whyItMatters":"The results provide a principled, information‑theoretic measure of parallel sampling efficiency that can guide the design of decoding rules for masked diffusion models and other generative systems."
arXiv:2609.16450v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) offer a promising parallel decoding paradigm as an alternative to autoregressive generation through iterative...
Diffusion language models enable flexible arbitrary-order generation, but existing sampling methods are mostly designed for early masked diffusion models (MDMs). In this work, we study sampling for recent block diffusion language models (BDLMs).
arXiv:2609.39560v1 Announce Type: new Abstract: Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional...
arXiv:2605. 13026v2 Announce Type: replace-cross Abstract: Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling.
arXiv:2602. 15008v2 Announce Type: replace Abstract: Diffusion models over discrete spaces have recently shown striking empirical success, yet their theoretical foundations remain incomplete.