arXiv AI September 1, 2026 By Injin Kong, Hyoungjoon Lee, Yohan Jo

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

The study investigates how post‑training of large autoregressive language models (ARMs) into masked diffusion models (MDMs) affects their internal computation. Across two 7B ARM‑MDM families and four diagnostic tasks, the authors find that MDMs retain much of the ARM’s high‑attribution pathways on prefix‑dominant tasks, but reorganize computation toward earlier layers on globally constrained tasks. Component‑level probes reveal that ARMs depend on sharply specialized components, whereas MDMs show weaker specialization and more diffuse output‑space alignment.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 8

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

arXiv:2606. 06712v1 Announce Type: cross Abstract: We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs).

By Xingyu Su, Jacob Helwig, Shubham Parashar, Atharv Chagi, Lakshmi Jotsna, Degui Zhi, James Caverlee, Dileep Kalathil, Shuiwang Ji

llms diffusion efficiency

arXiv Machine Learning

Jun 11

Teaching Diffusion to Speculate Left-to-Right

arXiv:2606. 11552v1 Announce Type: cross Abstract: Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation.

By Lexington Whalen, Yuki Ito, Ryo Sakamoto

llms diffusion benchmarks safety

Hugging Face Trending Papers

Oct 6

Enhancing Diffusion Language Models with Autoregressive Post-Training Weights

Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders and parallel decoding. Recent dLLMs are often in...

llms diffusion reinforcement-learning fine-tuning

arXiv AI

Sep 30

On Trajectory-Aware Training for Masked Diffusion Language Models

arXiv:2609.37974v1 Announce Type: cross Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The m...

By Manuel Madeira, Amitis Shidani, Alice Bizeul, Victor Turrisi, Louis B\'ethune, Bhavika Devnani, Dan Busbridge, Pierre Ablin, Jo\~ao Monteiro

llms diffusion fine-tuning safety

arXiv AI

Sep 3

Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models

The paper introduces PILL, a new infilling technique for diffusion language models that eliminates the need for a preset initial length and reduces inference overhead. PILL uses probing-based length-free decoding, cutting down on extra forward passes and speeding up generation. Experiments across five diffusion models and eight benchmarks show PILL outperforms the strongest baseline with higher pass rates and BLEU-2 scores while running 1.82× faster.

By Haobo Xu, Sirui Chen, Yuanchen Bei, Lingjie Chen, Yuchen Yan, Dongqi Fu, Jingrui He, Hanghang Tong

llms diffusion benchmarks

arXiv Machine Learning

Oct 7

Enhancing Diffusion Language Models with Autoregressive Post-Training Weights

arXiv:2610.08108v1 Announce Type: new Abstract: Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders...

By Yiming Qin, Ke Wang, Amel Abdelraheem, Adam Hazimeh, Pascal Frossard

llms diffusion reinforcement-learning fine-tuning

One email a morning, machine-written

Related stories

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

Teaching Diffusion to Speculate Left-to-Right

Enhancing Diffusion Language Models with Autoregressive Post-Training Weights

On Trajectory-Aware Training for Masked Diffusion Language Models

Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models

Enhancing Diffusion Language Models with Autoregressive Post-Training Weights