The paper introduces a framework that unifies diffusion and autoregressive (AR) generative models as paths on a corruption lattice, defining a cost for decoding schedules based on discarded parallel steps. It shows that the minimal number of steps for a zero‑cost schedule is determined by the data’s geometry—specifically, the graph’s treedepth for Markov data—providing a theoretical limit for both token and continuous field models. By estimating pairwise dependencies from pretrained weights, the authors predict the ranking of decoding schedules before execution, and validate these predictions across text, image, and video generation tasks.
By T. Y. Tsui, Jiatao Gu, Lingjie Liu
arXiv:2607. 01775v1 Announce Type: new Abstract: Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models.
By Marianne Arriola, Volodymyr Kuleshov
The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.
By Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis B\'ethune, Pierre Ablin, Marco Cuturi
The paper introduces Clock Diffusion, a framework for semi‑autoregressive continuous diffusion language models that incorporates position‑dependent noise schedules, efficient training, and sampling algorithms. It presents two generation modes—block and sliding window—and defines ClockDLMs, a family of Gaussian models that achieve state‑of‑the‑art diffusion likelihoods on OpenWebText and outperform continuous baselines on GSM8K while matching or exceeding discrete diffusion models. The authors also propose Cache Grab, a set of efficient samplers that adapt accelerated inference techniques from discrete diffusion to further improve model quality and efficiency.
By Yair Schiff, Omer Belhasin, Roy Uziel, Matan Rusanovsky, Ran Zilberstein, Marianne Arriola, Gilad Turok, Guanghan Wang, Volodymyr Kuleshov, Michael Elad
PlaidQ is a 0.7B continuous diffusion language model designed for code generation. By distilling its iterative refinement trajectory into only a few denoising steps—or even a single step—PlaidQ achieves competitive performance with discrete diffusion models while dramatically reducing inference time. The study demonstrates that continuous diffusion can be effectively compressed, enabling efficient and accurate code generation with minimal computational overhead.
By Fred Zhangzhi Peng, Kaiwen Zheng, Anru R. Zhang
arXiv:2607. 07026v1 Announce Type: new Abstract: Constrained decoding is essential for serving LLMs, ensuring that generated outputs follow specific structures such as JSON schema-formatted function calls.
By Meihua Dang, Stefano Ermon
Constrained decoding is essential for serving LLMs, ensuring that generated outputs follow specific structures such as JSON schema-formatted function calls. Existing systems are designed for autoregressive models and assume left-to-right generation, masking out invalid next tokens at each step.
Hierarchical Continuous Diffusion Language Models (HC‑DLM) combine discrete token generation with a continuous latent trajectory in a single denoising process, addressing the independence bottleneck of parallel decoding in discrete diffusion models. The training objective is derived from a variational bound on token likelihood, and the latent state is the sole persistent generative element, with tokens read out and fed back at each step. Experiments on Sudoku, Countdown, and LM1B show HC‑DLM outperforming both discrete and continuous diffusion baselines in puzzle accuracy and generative perplexity.
By Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing
The paper introduces dSpark, a schedule distillation framework that learns when to commit tokens in diffusion large language models to accelerate generation. By constructing unmasking schedules that prioritize revealing tokens which make other positions easier to predict, dSpark reduces uncertainty earlier in decoding. Experiments on mathematical reasoning and code generation tasks show significant speedups—10.12× on LLaDA‑8B‑Instruct and 5.12× on Dream‑7B‑Instruct—while preserving superior task performance.
By Aidar Myrzakhan, Zhiqiang Shen
arXiv:2607. 24507v1 Announce Type: cross Abstract: Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling.
By Xiaoyi Jiang, Jingyuan Li, Yixuan Jiang, Wei Liu, Yi Zhu, Zuoqiang Shi, Pipi Hu
arXiv:2610.22543v1 Announce Type: new
Abstract: Diffusion language models (DLMs) promise parallel text generation, but in practice face an unresolved speed-quality trade-off. Exact decoding fills in...
By Eliezer Shlomi, Eliron Rahimi, Avi Mendelson, Chaim Baskin
arXiv:2609. 04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation.
By Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing, Zhengzhong Liu