arXiv AI By Haoyang Zheng, Yidong Ouyang, Blake Jianhang Chen, Tian Tan, Ziqian Lin, Liqian Peng, Alec Go, Guang Lin

When Should Diffusion Language Models Reconsider Their Commitments?

Read the original on arXiv AI →

Masked Diffusion Models (MDMs) generate text quickly through parallel decoding but can commit to inconsistent predictions. The paper introduces PURL, a training framework that unifies parallel generation with selective token revision by learning a joint transition over edit selection, unmask selection, and token prediction. Experiments on logic puzzles, mathematical reasoning, and code generation show PURL improves accuracy by up to 47.4% and reduces decoding rounds by up to eightfold, while revealing that effective revisions depend on timing and context.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Nov 6

Corrective Diffusion Language Models

The paper introduces Corrective Diffusion Language Models (CDLMs), which enhance masked diffusion language models by explicitly supervising visible corrupted tokens to enable confidence-guided refinement of incorrect tokens. Experiments show that this correction-oriented post‑training improves iterative refinement and parallel decoding across models ranging from 0.5B to 8B, outperforming standard masked‑diffusion fine‑tuning on benchmarks such as Code Revision, HumanEval(+), and MATH. The authors also provide a new Code Revision Benchmark and release code and data for reproducibility.

By Shuibai Zhang, Fred Zhangzhi Peng, Yiheng Zhang, Jin Pan, Grigorios G. Chrysos
arXiv Computation and Language
Oct 1

DEdit: Iterative Draft Editing for Speculative Decoding

arXiv:2609.38510v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive LLMs by having a lightweight drafter propose tokens that the target model verifies in parallel. Diffusi...

By Longxuan Yu, Bingsen Chen, Peng Shi, Dongkyu Lee, Yi Xiang, Hideo Kobayashi, Sheng Zhang, Shuaichen Chang, Xing Niu, Zhuoyan Xu, Greg Ver Steeg, Jiarong Jiang
arXiv Machine Learning
Oct 19

LED: Improving Masked Diffusion Language Models by Lookahead Editing

The paper introduces Lookahead Editing (LED), a method that enhances masked diffusion language models (MDMs) by adding a rewritable proposal state. LED performs multiple lookahead updates at each decoding step to refine future predictions before committing tokens, thereby improving expressivity while retaining MDMs’ simple transition schedule. Fine‑tuned LED models (LED‑7B and LED‑8B) achieve significant performance gains on eight benchmarks and outperform larger models such as LLaDA2.1‑16B using only supervised fine‑tuning.

By Yuyan Zhou, Zeyuan Yang, Yuxuan Song, Zhicheng Cai, Hongli Yu, Weiying Ma, Hao Zhou, James Kwok
arXiv Computation and Language
Oct 28

HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models

HiLo-RD (High-to-Low Budget Representation Distillation) is a method designed to enhance masked diffusion language models (MDLMs) when operating under limited generation budgets. It pairs large-window teacher states with cropped small-window views and aligns their hidden representations at committed positions, guiding both token prediction and commitment order without an external teacher. Experiments on mathematics benchmarks and code generation show that HiLo-RD raises low-budget accuracy—e.g., a 9.4‑point gain on GSM8K—and reduces response length by about 25%.

By Xiang Li, Chengting Yu, Jungang Li, Yilun Zheng, Junzhuo Li, Xingyao Zhang, Wenbo Su, Xuming Hu