← Back to all news
arXiv Machine Learning By Ruurd J. A. Kuiper, Robert A. Bagheri

BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion

Read the original on arXiv Machine Learning →

The paper introduces BYOD (Build Your Own DLM), a LoRA‑only method that converts autoregressive language models into diffusion language models without changing the base architecture. BYOD successfully converts Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B, adding adapters that occupy only 4.1‑5.8% of the total parameters while keeping all base‑model weights frozen. The conversions require at most 102.4 M tokens and a few hours on a single GPU, and the resulting diffusion models can generate from fully masked sequences, use bidirectional context, and retain much of the original model’s knowledge, with the Gemma conversion achieving the highest accuracy on a nine‑task benchmark.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 4

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

arXiv:2609. 04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation.

By Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing, Zhengzhong Liu
llmsagentsdiffusionefficiencybenchmarks
More like this →
arXiv Machine Learning
Oct 5

Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

The paper compares two low‑budget methods for converting a 30B Mixture‑of‑Experts autoregressive language model into a diffusion language model. One method updates a subset of the model’s weights in‑place, while the other freezes the context tower and conditions on a frozen causal copy via cross‑attention. With only 1B training tokens, the frozen‑tower approach achieves a HumanEval pass@10 score of 71.60 versus 6.19 for the in‑place method, and retains 95% of the parent’s GSM8K and 99% of its MMLU‑Pro performance.

By Wentao Lu, Jesse Clark, Tianyu Zhu
llmsdiffusionbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 16

On-Policy Residual Learning for Diffusion Language Models

On-Policy Residual Learning for Diffusion Language Models (ORL) introduces a framework that learns residual context directly from model-generated trajectories, using an entropy-contraction objective to sharpen predictions. The approach co-designs algorithmic and system components—such as asynchronous residual capture, block-diffusion attention, and packed compute-balanced training—to enable practical scaling. Experiments on a 4B DLM converted from Qwen3-4B show up to 6.2× faster tokens per forward and 6.9× higher end-to-end throughput, while also improving speed-quality trade-offs over autoregressive models and transferring effectively to other DLMs like LLaDA-2.1-mini and DiffusionGemma.

By Yifan Yu, Yuqing Jian, Jisen Li, Yuezhou Hu, Tianyi Zhang, Xiaoxia Wu, Rui-Jie Zhu, Junxiong Wang, Zhongzhu Zhou, Fan Lai, James Zou, Chenfeng Xu
llmsdiffusionefficiency
More like this →