arXiv Machine Learning
Jun 17
Recursive Scaling in Masked Diffusion Models
arXiv:2606. 18022v1 Announce Type: new Abstract: Masked diffusion models (MDMs) have recently emerged as a promising paradigm for sequence generation.
Random Remasking Scales Masked Diffusion Inference
The paper introduces Random Remask-Denoise (R$^2$D) inference, a test‑time, finetuning‑free method for masked diffusion models that randomly re‑introduces masks and denoises to form a Markov chain on clean sequences. It also proposes Trajectory Correction Training (TCT), which trains the denoiser by exposing it to its own errors during generation. Experiments show that R$^2$D with TCT achieves near‑log‑linear scaling, achieving over 97% accuracy on hard Sudoku puzzles, over 99% on 3‑SAT, and surpasses iterative correction baselines on language tasks such as HumanEval+.
Related stories
arXiv Machine Learning
Jun 17
Learn from Your Mistakes: Self-Correcting Masked Diffusion Models
arXiv:2602. 11590v3 Announce Type: replace Abstract: Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models, enabling parallel token generation while achieving competitive performance.
arXiv AI
Sep 30
On Trajectory-Aware Training for Masked Diffusion Language Models
arXiv:2609.37974v1 Announce Type: cross Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The m...
arXiv Computation and Language
Aug 21
Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Language Models
arXiv:2604. 18738v3 Announce Type: replace Abstract: Diffusion language models (dLLMs) generate text through iterative denoising, filling multiple masked positions at each step.
arXiv Machine Learning
Oct 22
Looped Uniform Diffusion Language Models
The paper introduces looped uniform diffusion language models (UDLMs), which iteratively denoise token sequences and improve training efficiency by applying a few early Transformer layers repeatedly within each denoising step. Experiments show that looped UDLMs achieve the same validation negative log‑likelihood as a non‑looped model trained on roughly half the tokens, while reducing training FLOPs by up to 2.8× on OpenWebText and 3.3× on FineWeb‑Edu. Additionally, with a matched number of denoising steps, looped UDLMs attain up to 9.8% lower generation perplexity, and sampling smaller loop counts more often during training further boosts inference efficiency without sacrificing accuracy.
arXiv Machine Learning
Jun 29
VGB for Masked Diffusion Model: Efficient Test-time Scaling for Reward Satisfaction and Sample Editing
arXiv:2606. 28301v1 Announce Type: new Abstract: Inference-time scaling is a promising paradigm to improve generative models, especially when outputs must satisfy structural constraints or optimize downstream rewards.
arXiv AI
Sep 10
Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks
The paper examines how Denoising Diffusion Probabilistic Models (DDPMs) perform on globally constrained discrete tasks such as Sudoku and N-queens. It shows that standard diffusion sampling, which keeps updates close to the noisy state, often preserves early mistakes, whereas sampling directly from the model’s clean predictions dramatically improves validity (e.g., Sudoku from 31% to 95%). The authors further introduce self‑correction training, exposing the model to its own predictions to reduce inference errors, which enhances the performance of standard samplers across tasks.
arXiv AI
Oct 15
RL Post-Training Diffusion Language Models Along Their Own Generation Paths
arXiv:2610.17266v1 Announce Type: new Abstract: Diffusion language models are a promising and more efficient alternative to autoregressive models. However, most RL post-training techniques are not ea...
arXiv Machine Learning
Jun 5
Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
arXiv:2602. 10314v2 Announce Type: replace Abstract: Masked Diffusion Models (MDMs) have emerged as a promising approach for generative modeling in discrete spaces.
arXiv AI
Oct 15
Correction Training for Diffusion Models
arXiv:2610.14136v2 Announce Type: new Abstract: Iterative sampling in diffusion models can lead to error accumulation due to exposure bias. The denoiser is trained on real images interpolated with no...
arXiv Machine Learning
Oct 8
CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models
CoDR (Confidence Drift Remasking) is a training‑free, sampler‑agnostic refinement pass for masked diffusion language models that detects and corrects early token commitments that become unsupported as context grows denser. It estimates confidence drift for all committed positions using only a few forward passes and remasks only the tokens whose confidence has dropped, thereby improving accuracy across multiple backbones, tasks, and samplers with modest computational overhead.
arXiv Computation and Language
Sep 23
Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models
Informed Masking (IM) is a new technique for aligning Diffusion Large Language Models (dLLMs) with Reinforcement Learning (RL). It identifies a systematic upstream/downstream token structure in dLLM rollouts and shows that masking downstream tokens creates better subproblems for likelihood estimation. When integrated into three state‑of‑the‑art dLLM RL methods on LLaDA‑8B‑Instruct, IM yields up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks while improving training stability.