The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Diffusion language models (DLMs) enable flexible parallel generation by resolving tokens according to the model's evolving predictability rather than a fixed left-to-right order. This flexibility, how...
The paper introduces Predictability-Aware Canvas Selection and Supervision (PACS), a supervised fine‑tuning method for diffusion language models that aligns training with inference by using the model’s own predictability to determine which tokens are visible and when they are supervised. PACS selects the attention‑visible input length for each example and supervises reference tokens just before they become confident, addressing mismatches caused by random masking. Experiments on reasoning and code‑generation benchmarks show PACS consistently outperforms standard SFT, achieving up to 12.84 percentage‑point gains on individual tasks.
The paper introduces dSpark, a schedule distillation framework that learns when to commit tokens in diffusion large language models to accelerate generation. By constructing unmasking schedules that prioritize revealing tokens which make other positions easier to predict, dSpark reduces uncertainty earlier in decoding. Experiments on mathematical reasoning and code generation tasks show significant speedups—10.12× on LLaDA‑8B‑Instruct and 5.12× on Dream‑7B‑Instruct—while preserving superior task performance.
arXiv:2606. 06712v1 Announce Type: cross Abstract: We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs).
The paper introduces PILL, a new infilling technique for diffusion language models that eliminates the need for a preset initial length and reduces inference overhead. PILL uses probing-based length-free decoding, cutting down on extra forward passes and speeding up generation. Experiments across five diffusion models and eight benchmarks show PILL outperforms the strongest baseline with higher pass rates and BLEU-2 scores while running 1.82× faster.
The paper introduces Lookahead Editing (LED), a method that enhances masked diffusion language models (MDMs) by adding a rewritable proposal state. LED performs multiple lookahead updates at each decoding step to refine future predictions before committing tokens, thereby improving expressivity while retaining MDMs’ simple transition schedule. Fine‑tuned LED models (LED‑7B and LED‑8B) achieve significant performance gains on eight benchmarks and outperform larger models such as LLaDA2.1‑16B using only supervised fine‑tuning.