The paper introduces Predictability-Aware Canvas Selection and Supervision (PACS), a supervised fine‑tuning method for diffusion language models that aligns training with inference by using the model’s own predictability to determine which tokens are visible and when they are supervised. PACS selects the attention‑visible input length for each example and supervises reference tokens just before they become confident, addressing mismatches caused by random masking. Experiments on reasoning and code‑generation benchmarks show PACS consistently outperforms standard SFT, achieving up to 12.84 percentage‑point gains on individual tasks.
By Miao Li, Hanyang Jiang, Hengyu Fu, Baihe Huang, Yating Liu, Yuhang Cai, Changkun Guan, Zhongzhu Chen, Jiecheng Lu, Michael A. Boateng, Pascal Van Hentenryck
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation. Speculative decoding addresses this bottleneck by employing a lightweight draft model to propose multiple future tokens that are subsequently verified in parallel by a larger target model.
The paper introduces PILL, a new infilling technique for diffusion language models that eliminates the need for a preset initial length and reduces inference overhead. PILL uses probing-based length-free decoding, cutting down on extra forward passes and speeding up generation. Experiments across five diffusion models and eight benchmarks show PILL outperforms the strongest baseline with higher pass rates and BLEU-2 scores while running 1.82× faster.
By Haobo Xu, Sirui Chen, Yuanchen Bei, Lingjie Chen, Yuchen Yan, Dongqi Fu, Jingrui He, Hanghang Tong
arXiv:2609.38385v1 Announce Type: new
Abstract: Supervised fine-tuning of discrete diffusion language models masks some response tokens and trains the model to recover their original values from the...
By Loay Mualem, Llu\'is Pastor-P\'erez, Vinh Tong, Andrei Manolache, Tanja Bien, Steffen Staab, Mathias Niepert
arXiv:2606. 11552v1 Announce Type: cross Abstract: Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation.
By Lexington Whalen, Yuki Ito, Ryo Sakamoto
arXiv:2606. 25331v1 Announce Type: cross Abstract: Modern large language models are predominantly trained with autoregressive factorization and causal attention.
By Shen Nie, Qiyang Min, Shaoxuan Xu, Zihao Huang, Yuxuan Song, Yong Shan, Yankai Lin, Wayne Xin Zhao, Chongxuan Li, Ji-Rong Wen
arXiv:2609.37974v1 Announce Type: cross
Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The m...
By Manuel Madeira, Amitis Shidani, Alice Bizeul, Victor Turrisi, Louis B\'ethune, Bhavika Devnani, Dan Busbridge, Pierre Ablin, Jo\~ao Monteiro
arXiv:2606. 19475v1 Announce Type: new Abstract: Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks.
By Thomas Bertolani, Davide Bucciarelli, Leonardo Zini, Marcella Cornia, Lorenzo Baraldi
arXiv:2604. 18995v2 Announce Type: replace-cross Abstract: Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregressive generation by enabling parallel token prediction.
By Zhenbang Du, Kejing Xia, Xinrui Zhong, Yonggan Fu, Nicolai Oswald, Binfei Ji, Brucek Khailany, Pavlo Molchanov, Yingyan Lin
arXiv:2608.30922v1 Announce Type: new
Abstract: Masked diffusion language models predict tokens from a partially observed response canvas, enabling bidirectional conditioning and parallel token refin...
By Wail Bouhedja, Amr Mohamed, Guokan Shang
Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation. A notable limitation of the masked formulation, however, is that once a token has been unmasked it can no longer be revised, leaving dLLMs vulnerable to early sampling mistakes.
arXiv:2601.03199v2 Announce Type: replace-cross
Abstract: Diffusion language models (DLMs) have shown strong potential for general natural language tasks with in-context examples. Existing In-Context...
By Yang Li, Han Meng, Chenan Wang, Zhenyu Bi, Xuan Wang, Haipeng Chen