The paper introduces BYOD (Build Your Own DLM), a LoRA‑only method that converts autoregressive language models into diffusion language models without changing the base architecture. BYOD successfully converts Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B, adding adapters that occupy only 4.1‑5.8% of the total parameters while keeping all base‑model weights frozen. The conversions require at most 102.4 M tokens and a few hours on a single GPU, and the resulting diffusion models can generate from fully masked sequences, use bidirectional context, and retain much of the original model’s knowledge, with the Gemma conversion achieving the highest accuracy on a nine‑task benchmark.
arXiv:2609. 04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation.
The paper compares two low‑budget methods for converting a 30B Mixture‑of‑Experts autoregressive language model into a diffusion language model. One method updates a subset of the model’s weights in‑place, while the other freezes the context tower and conditions on a frozen causal copy via cross‑attention. With only 1B training tokens, the frozen‑tower approach achieves a HumanEval pass@10 score of 71.60 versus 6.19 for the in‑place method, and retains 95% of the parent’s GSM8K and 99% of its MMLU‑Pro performance.
arXiv:2610.17590v1 Announce Type: cross Abstract: Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to t...
arXiv:2608.22646v1 Announce Type: new Abstract: Diffusion language models can generate many tokens in parallel, but they still require repeated denoising steps during inference. This makes generation...
On-Policy Residual Learning for Diffusion Language Models (ORL) introduces a framework that learns residual context directly from model-generated trajectories, using an entropy-contraction objective to sharpen predictions. The approach co-designs algorithmic and system components—such as asynchronous residual capture, block-diffusion attention, and packed compute-balanced training—to enable practical scaling. Experiments on a 4B DLM converted from Qwen3-4B show up to 6.2× faster tokens per forward and 6.9× higher end-to-end throughput, while also improving speed-quality trade-offs over autoregressive models and transferring effectively to other DLMs like LLaDA-2.1-mini and DiffusionGemma.
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
The paper introduces PACE-dLLM, an acceleration method for diffusion language models (dLLMs) that uses the model’s own per‑step confidence to estimate a ‘confidence cliff’ and determine the optimal look‑ahead horizon for block decoding. By fitting this cliff in closed form at each step, PACE-dLLM sets the horizon to its saturation point and applies an independent confidence threshold for token commitment, thereby avoiding the trade‑offs inherent in fixed‑size block decoding. Experiments on reasoning and code benchmarks show that PACE-dLLM achieves the best average accuracy on open‑source dLLM backbones while delivering significant wall‑clock speedups—up to 5.23× on LLaDA and 3.06× on Dream—improving the quality‑throughput Pareto frontier.
arXiv:2608. 03457v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood.
arXiv:2606. 01774v1 Announce Type: cross Abstract: Autoregressive (AR) large language models (LLMs) have achieved broad practical success, but sequential decoding remains a key bottleneck for low-latency deployment.
arXiv:2607. 24507v1 Announce Type: cross Abstract: Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling.
arXiv:2511. 15927v4 Announce Type: replace-cross Abstract: Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead.
The paper introduces Lookahead Editing (LED), a method that enhances masked diffusion language models (MDMs) by adding a rewritable proposal state. LED performs multiple lookahead updates at each decoding step to refine future predictions before committing tokens, thereby improving expressivity while retaining MDMs’ simple transition schedule. Fine‑tuned LED models (LED‑7B and LED‑8B) achieve significant performance gains on eight benchmarks and outperform larger models such as LLaDA2.1‑16B using only supervised fine‑tuning.