arXiv Computation and Language By Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Read the original on arXiv Computation and Language →

LangFlow is a continuous diffusion language model that bridges embedding-space diffusion with Flow Matching, introducing three key innovations: an ODE-based NLL bound, an information-uniform noise schedule using a learnable Gumbel distribution, and self-conditioning training. These advances enable LangFlow to match or surpass top discrete diffusion models in generative quality, achieving perplexities of 92.3 on LM1B and 36.5 on OpenWebText, and outperforming discrete counterparts on reasoning tasks such as Sudoku. The work demonstrates that continuous diffusion can rival discrete diffusion for language modeling, offering a promising new paradigm.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Oct 2

Hierarchical Continuous Diffusion Language Models

Hierarchical Continuous Diffusion Language Models (HC‑DLM) combine discrete token generation with a continuous latent trajectory in a single denoising process, addressing the independence bottleneck of parallel decoding in discrete diffusion models. The training objective is derived from a variational bound on token likelihood, and the latent state is the sole persistent generative element, with tokens read out and fed back at each step. Experiments on Sudoku, Countdown, and LM1B show HC‑DLM outperforming both discrete and continuous diffusion baselines in puzzle accuracy and generative perplexity.

By Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing
arXiv Machine Learning
Oct 7

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

The paper introduces Hierarchical Continuous Diffusion Language Models (H-CDLMs), a framework that jointly diffuses multiple token modalities—individual tokens and coarser clusters of token embeddings—to enhance continuous diffusion language models. Applied to the CoBit architecture, the resulting H-CoBit achieves significant empirical gains, improving MAUVE scores and achieving lower generative perplexity on LM1B and OWT, while also outperforming prior continuous diffusion models on GSM8K. The approach generalizes to other continuous generative paradigms, as shown by consistent improvements when applied to the flow matching model FLM.

By Mathias Ollu, Nikos Komodakis
arXiv Machine Learning
Oct 12

On the Design Space of Continuous Diffusion Language Models

The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.

By Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis B\'ethune, Pierre Ablin, Marco Cuturi
arXiv AI
Oct 5

Large Language Continuous Diffusion Models

The paper introduces Sigma, a large-scale continuous diffusion language model (3B/8B parameters) that uses steerable, low-dimensional ODE/SDE latent trajectories to address the non-smoothness of discrete diffusion models. Sigma is trained blockwise via likelihood optimization, jointly denoises Gaussian-corrupted token embeddings, and learns an optimal embedding geometry, leveraging pre-trained autoregressive weights for faster training. During inference, classifier-free guidance and score temperature are identified as essential for high-fidelity reasoning and coding, and Sigma matches or exceeds discrete models on benchmarks such as GSM8K, Minerva, HumanEval, MBPP, MATH-500, and AIME, while also revealing unique structural benefits like embedding-space steering and graceful degradation for low NFEs.

By Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Juki\'c, Arash Vahdat, Morteza Mardani
arXiv Computation and Language
Nov 5

AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

AURORA-LM is a continuous‑latent diffusion language model that separates the creation of a high‑capacity, decodable text representation from the modeling of its distribution. It uses a Query‑based Encoder‑Decoder to produce a prefix‑aligned latent sequence and a Block‑Causal Diffusion Transformer to learn its distribution via flow matching, generating text block‑by‑block while denoising positions in parallel. The model calibrates noise levels to latent width, introduces self‑trajectory consistency, and achieves state‑of‑the‑art performance on OpenWebText generation and XSum summarization, outperforming larger publicly released latent‑diffusion models when scaled to 1 B parameters.

By Jiajun Liang, Yucheng Liao, Yukang Cao, Jiazhe Wei, Ken Li, Wende Tan, Jiankun Zhang, ZY Cui, Jingkang Yang, Liucheng Guo, Shiqi Yang, B. Yang, Caifeng Shan, Ziwei Liu, Chenyang Si
arXiv Machine Learning
Sep 11

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

The paper revisits the continuous diffusion language model Plaid and introduces RePlaid, aligning its architecture with modern discrete diffusion models. RePlaid achieves a compute gap of only 20× compared to autoregressive models, surpasses Duo with fewer parameters, and outperforms MDLM in over‑trained settings. On OpenWebText, RePlaid sets a new state‑of‑the‑art continuous diffusion perplexity of 22.1 and demonstrates superior generation quality, while theoretical analysis links likelihood‑based training to linear cross‑entropy over time and structured embedding geometries.

By Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun