The paper introduces Corrective Diffusion Language Models (CDLMs), which enhance masked diffusion language models by explicitly supervising visible corrupted tokens to enable confidence-guided refinement of incorrect tokens. Experiments show that this correction-oriented post‑training improves iterative refinement and parallel decoding across models ranging from 0.5B to 8B, outperforming standard masked‑diffusion fine‑tuning on benchmarks such as Code Revision, HumanEval(+), and MATH. The authors also provide a new Code Revision Benchmark and release code and data for reproducibility.
By Shuibai Zhang, Fred Zhangzhi Peng, Yiheng Zhang, Jin Pan, Grigorios G. Chrysos
arXiv:2609.38510v1 Announce Type: new
Abstract: Speculative decoding accelerates autoregressive LLMs by having a lightweight drafter propose tokens that the target model verifies in parallel. Diffusi...
By Longxuan Yu, Bingsen Chen, Peng Shi, Dongkyu Lee, Yi Xiang, Hideo Kobayashi, Sheng Zhang, Shuaichen Chang, Xing Niu, Zhuoyan Xu, Greg Ver Steeg, Jiarong Jiang
The paper introduces Lookahead Editing (LED), a method that enhances masked diffusion language models (MDMs) by adding a rewritable proposal state. LED performs multiple lookahead updates at each decoding step to refine future predictions before committing tokens, thereby improving expressivity while retaining MDMs’ simple transition schedule. Fine‑tuned LED models (LED‑7B and LED‑8B) achieve significant performance gains on eight benchmarks and outperform larger models such as LLaDA2.1‑16B using only supervised fine‑tuning.
By Yuyan Zhou, Zeyuan Yang, Yuxuan Song, Zhicheng Cai, Hongli Yu, Weiying Ma, Hao Zhou, James Kwok
HiLo-RD (High-to-Low Budget Representation Distillation) is a method designed to enhance masked diffusion language models (MDLMs) when operating under limited generation budgets. It pairs large-window teacher states with cropped small-window views and aligns their hidden representations at committed positions, guiding both token prediction and commitment order without an external teacher. Experiments on mathematics benchmarks and code generation show that HiLo-RD raises low-budget accuracy—e.g., a 9.4‑point gain on GSM8K—and reduces response length by about 25%.
By Xiang Li, Chengting Yu, Jungang Li, Yilun Zheng, Junzhuo Li, Xingyao Zhang, Wenbo Su, Xuming Hu
arXiv:2606. 11552v1 Announce Type: cross Abstract: Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation.
By Lexington Whalen, Yuki Ito, Ryo Sakamoto
arXiv:2604. 18738v3 Announce Type: replace Abstract: Diffusion language models (dLLMs) generate text through iterative denoising, filling multiple masked positions at each step.
By Lin Yao
arXiv:2610.34310v1 Announce Type: cross
Abstract: Insertion-based masked diffusion models enable flexible-length, parallel generation by interleaving mask insertion with token unmasking. However, bec...
By Durga Prasad Maram, Dhruvesh Patel, Tahira Naseem, Benjamin Rozonoyer, Gaurav Pandey, Ram\'on Fernandez Astudillo, Andrew McCallum
The paper introduces dSpark, a schedule distillation framework that learns when to commit tokens in diffusion large language models to accelerate generation. By constructing unmasking schedules that prioritize revealing tokens which make other positions easier to predict, dSpark reduces uncertainty earlier in decoding. Experiments on mathematical reasoning and code generation tasks show significant speedups—10.12× on LLaDA‑8B‑Instruct and 5.12× on Dream‑7B‑Instruct—while preserving superior task performance.
By Aidar Myrzakhan, Zhiqiang Shen
CoDR (Confidence Drift Remasking) is a training‑free, sampler‑agnostic refinement pass for masked diffusion language models that detects and corrects early token commitments that become unsupported as context grows denser. It estimates confidence drift for all committed positions using only a few forward passes and remasks only the tokens whose confidence has dropped, thereby improving accuracy across multiple backbones, tasks, and samplers with modest computational overhead.
By Yue Wu, Qinghe Zhang, Yu Zhang, Jian Huang
The paper investigates whether the locate‑then‑edit approach for knowledge editing, previously applied only to autoregressive language models, can be transferred to masked diffusion models (MDMs). It finds that the optimal edit location—an early‑to‑mid‑layer MLP at the last subject token—remains the same for both model types, but that MDMs suffer a sharper decline in performance when editing longer, multi‑token facts. By incorporating intermediate partially‑unmasked states into the edit optimization, the authors restore multi‑token editing performance in MDMs.
By Haewon Park, Yohan Jo
The paper introduces Reliable Parallel Decoding (RPD) for masked diffusion language models, addressing the unreliability of committing multiple predictions from a single forward pass. Diagnostics reveal that confidence alone is insufficient, as confident end‑sequence predictions can preempt necessary upstream computations, and downstream predictions degrade with upstream uncertainty. RPD selects candidates based on layer‑wise stability and final confidence, committing them under an entropy budget while deferring uncertain predictions, achieving superior throughput and competitive accuracy on LLaDA and Dream benchmarks.
By Zhenghao He, Bohan Liu, Guangzhi Xiong, Aidong Zhang
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation. Speculative decoding addresses this bottleneck by employing a lightweight draft model to propose multiple future tokens that are subsequently verified in parallel by a larger target model.