← Back to all news
arXiv Machine Learning By Jinjin He, Xinsong Feng, Lucheng Fu, Sinan Wang, Bo Zhu, Jiayun Wang

Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation

Read the original on arXiv Machine Learning →

The paper proposes Predictive-State Flows (PSF), a new architecture for continuous diffusion language models that separates the diffusion prior’s role to generate high‑level predictive states and the decoder’s role to resolve token‑level choices. PSF partitions sequences into blocks, uses a rectified‑flow prior to produce predictive states, and then a block‑causal decoder generates all blocks in parallel, reducing token‑level sequential depth. Experiments on OpenWebText show that a 197M‑parameter PSF model with eight predictive states achieves a generative perplexity of 22.8 and a MAUVE@512 of 0.93 while using far fewer FLOPs than a comparable ELF‑B model.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 7

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

The paper introduces Hierarchical Continuous Diffusion Language Models (H-CDLMs), a framework that jointly diffuses multiple token modalities—individual tokens and coarser clusters of token embeddings—to enhance continuous diffusion language models. Applied to the CoBit architecture, the resulting H-CoBit achieves significant empirical gains, improving MAUVE scores and achieving lower generative perplexity on LM1B and OWT, while also outperforming prior continuous diffusion models on GSM8K. The approach generalizes to other continuous generative paradigms, as shown by consistent improvements when applied to the flow matching model FLM.

By Mathias Ollu, Nikos Komodakis
llmsragdiffusionbenchmarks
More like this →
arXiv Computation and Language
Oct 12

Contextual Correction of Parallel Draftsfor Speculative Decoding

The paper introduces DCOCO (DFlash Contextual Corrector), a lightweight mechanism that restores intra-block dependencies in speculative decoding by evaluating multiple candidate tokens for contextual compatibility across adjacent draft positions. DCOCO maintains parallel drafting while enabling contextually informed token selection at low additional cost. Across multiple verifier backbones and benchmarks, DCOCO achieves the highest average throughput, delivering speedups of up to 6.6× over vanilla decoding and outperforming competing methods by 6–23% under concurrent serving loads.

By Roy Eisenstadt, Harel Lidar, Tal Gonen, Omer Luxembourg, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman
llmsdiffusionefficiencybenchmarks
More like this →