arXiv AI

When Should Diffusion Language Models Reconsider Their Commitments?

Masked Diffusion Models (MDMs) generate text quickly through parallel decoding but can commit to inconsistent predictions. The paper introduces PURL, a training framework that unifies parallel generation with selective token revision by learning a joint transition over edit selection, unmask selection, and token prediction. Experiments on logic puzzles, mathematical reasoning, and code generation show PURL improves accuracy by up to 47.4% and reduces decoding rounds by up to eightfold, while revealing that effective revisions depend on timing and context.

arXiv Machine Learning
Nov 6

Corrective Diffusion Language Models

The paper introduces Corrective Diffusion Language Models (CDLMs), which enhance masked diffusion language models by explicitly supervising visible corrupted tokens to enable confidence-guided refinement of incorrect tokens. Experiments show that this correction-oriented post‑training improves iterative refinement and parallel decoding across models ranging from 0.5B to 8B, outperforming standard masked‑diffusion fine‑tuning on benchmarks such as Code Revision, HumanEval(+), and MATH. The authors also provide a new Code Revision Benchmark and release code and data for reproducibility.

By Shuibai Zhang, Fred Zhangzhi Peng, Yiheng Zhang, Jin Pan, Grigorios G. Chrysos
arXiv Computation and Language
Oct 1

DEdit: Iterative Draft Editing for Speculative Decoding

arXiv:2609.38510v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive LLMs by having a lightweight drafter propose tokens that the target model verifies in parallel. Diffusi...

By Longxuan Yu, Bingsen Chen, Peng Shi, Dongkyu Lee, Yi Xiang, Hideo Kobayashi, Sheng Zhang, Shuaichen Chang, Xing Niu, Zhuoyan Xu, Greg Ver Steeg, Jiarong Jiang
arXiv Machine Learning
Oct 19

LED: Improving Masked Diffusion Language Models by Lookahead Editing

The paper introduces Lookahead Editing (LED), a method that enhances masked diffusion language models (MDMs) by adding a rewritable proposal state. LED performs multiple lookahead updates at each decoding step to refine future predictions before committing tokens, thereby improving expressivity while retaining MDMs’ simple transition schedule. Fine‑tuned LED models (LED‑7B and LED‑8B) achieve significant performance gains on eight benchmarks and outperform larger models such as LLaDA2.1‑16B using only supervised fine‑tuning.

By Yuyan Zhou, Zeyuan Yang, Yuxuan Song, Zhicheng Cai, Hongli Yu, Weiying Ma, Hao Zhou, James Kwok
arXiv Computation and Language
Oct 28

HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models

HiLo-RD (High-to-Low Budget Representation Distillation) is a method designed to enhance masked diffusion language models (MDLMs) when operating under limited generation budgets. It pairs large-window teacher states with cropped small-window views and aligns their hidden representations at committed positions, guiding both token prediction and commitment order without an external teacher. Experiments on mathematics benchmarks and code generation show that HiLo-RD raises low-budget accuracy—e.g., a 9.4‑point gain on GSM8K—and reduces response length by about 25%.

By Xiang Li, Chengting Yu, Jungang Li, Yilun Zheng, Junzhuo Li, Xingyao Zhang, Wenbo Su, Xuming Hu
arXiv AI
Oct 27

Learning When to Commit: Schedule Distillation for Fast Diffusion Language Models

The paper introduces dSpark, a schedule distillation framework that learns when to commit tokens in diffusion large language models to accelerate generation. By constructing unmasking schedules that prioritize revealing tokens which make other positions easier to predict, dSpark reduces uncertainty earlier in decoding. Experiments on mathematical reasoning and code generation tasks show significant speedups—10.12× on LLaDA‑8B‑Instruct and 5.12× on Dream‑7B‑Instruct—while preserving superior task performance.

By Aidar Myrzakhan, Zhiqiang Shen
arXiv Machine Learning
Oct 8

CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models

CoDR (Confidence Drift Remasking) is a training‑free, sampler‑agnostic refinement pass for masked diffusion language models that detects and corrects early token commitments that become unsupported as context grows denser. It estimates confidence drift for all committed positions using only a few forward passes and remasks only the tokens whose confidence has dropped, thereby improving accuracy across multiple backbones, tasks, and samplers with modest computational overhead.

By Yue Wu, Qinghe Zhang, Yu Zhang, Jian Huang
arXiv Computation and Language
Sep 3

Knowledge Editing for Masked Diffusion Language Models

The paper investigates whether the locate‑then‑edit approach for knowledge editing, previously applied only to autoregressive language models, can be transferred to masked diffusion models (MDMs). It finds that the optimal edit location—an early‑to‑mid‑layer MLP at the last subject token—remains the same for both model types, but that MDMs suffer a sharper decline in performance when editing longer, multi‑token facts. By incorporating intermediate partially‑unmasked states into the edit optimization, the authors restore multi‑token editing performance in MDMs.

By Haewon Park, Yohan Jo
arXiv AI
Sep 30

Reliable Parallel Decoding in Masked Diffusion Language Models

The paper introduces Reliable Parallel Decoding (RPD) for masked diffusion language models, addressing the unreliability of committing multiple predictions from a single forward pass. Diagnostics reveal that confidence alone is insufficient, as confident end‑sequence predictions can preempt necessary upstream computations, and downstream predictions degrade with upstream uncertainty. RPD selects candidates based on layer‑wise stability and final confidence, committing them under an entropy budget while deferring uncertain predictions, achieving superior throughput and competitive accuracy on LLaDA and Dream benchmarks.

By Zhenghao He, Bohan Liu, Guangzhi Xiong, Aidong Zhang
Hugging Face Trending Papers
Jun 10

Teaching Diffusion to Speculate Left-to-Right

Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation. Speculative decoding addresses this bottleneck by employing a lightweight draft model to propose multiple future tokens that are subsequently verified in parallel by a larger target model.