← Back to all news
arXiv Machine Learning By Zhaoyi Zhou, Frank Wu, Andrea Zanette

Random Remasking Scales Masked Diffusion Inference

Read the original on arXiv Machine Learning →

The paper introduces Random Remask-Denoise (R$^2$D) inference, a test‑time, finetuning‑free method for masked diffusion models that randomly re‑introduces masks and denoises to form a Markov chain on clean sequences. It also proposes Trajectory Correction Training (TCT), which trains the denoiser by exposing it to its own errors during generation. Experiments show that R$^2$D with TCT achieves near‑log‑linear scaling, achieving over 97% accuracy on hard Sudoku puzzles, over 99% on 3‑SAT, and surpasses iterative correction baselines on language tasks such as HumanEval+.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 22

Looped Uniform Diffusion Language Models

The paper introduces looped uniform diffusion language models (UDLMs), which iteratively denoise token sequences and improve training efficiency by applying a few early Transformer layers repeatedly within each denoising step. Experiments show that looped UDLMs achieve the same validation negative log‑likelihood as a non‑looped model trained on roughly half the tokens, while reducing training FLOPs by up to 2.8× on OpenWebText and 3.3× on FineWeb‑Edu. Additionally, with a matched number of denoising steps, looped UDLMs attain up to 9.8% lower generation perplexity, and sampling smaller loop counts more often during training further boosts inference efficiency without sacrificing accuracy.

By Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji, Praneeth Vepakomma
llmsdiffusion
More like this →