arXiv AI
Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 4
A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
arXiv:2603. 07475v4 Announce Type: replace-cross Abstract: Autoregressive (AR) language models build representations incrementally via left-to-right prediction, while diffusion language models (dLLMs) are trained through full-sequence denoising.
arXiv AI
Sep 3
Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models
The paper introduces PILL, a new infilling technique for diffusion language models that eliminates the need for a preset initial length and reduces inference overhead. PILL uses probing-based length-free decoding, cutting down on extra forward passes and speeding up generation. Experiments across five diffusion models and eight benchmarks show PILL outperforms the strongest baseline with higher pass rates and BLEU-2 scores while running 1.82× faster.
arXiv Machine Learning
Aug 10
Retrofitting Linear Attention into Diffusion Language Models
arXiv:2608. 06628v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer a promising alternative to autoregressive models by accelerating inference through parallel decoding.
arXiv AI
Jun 19
Diffusion Language Models: An Experimental Analysis
arXiv:2606. 19475v1 Announce Type: new Abstract: Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks.
arXiv Machine Learning
Sep 4
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
arXiv:2609. 04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation.
arXiv AI
Sep 16
Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling
arXiv:2609.16450v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) offer a promising parallel decoding paradigm as an alternative to autoregressive generation through iterative...