← Back to all news
arXiv AI By Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, Tianyang Hu

Designing Transformers for Continuous Diffusion Language Models

Read the original on arXiv AI →

The paper introduces the Language Diffusion Transformer (LangDiT), a new transformer architecture tailored for continuous diffusion language models (CDLMs). By incorporating feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing, calibration, and confidence-gated training self-conditioning, LangDiT achieves lower generation perplexity than existing models across various CDLM formulations. Experiments on the OpenWebText dataset show state-of-the-art perplexity scores, demonstrating the architecture’s effectiveness while preserving the underlying model representations and training objectives.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
1d ago

Looped Uniform Diffusion Language Models

arXiv:2610.30066v1 Announce Type: cross Abstract: Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupt...

By Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji, Praneeth Vepakomma
llmsdiffusion
More like this →
arXiv Machine Learning
Oct 7

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

The paper introduces Hierarchical Continuous Diffusion Language Models (H-CDLMs), a framework that jointly diffuses multiple token modalities—individual tokens and coarser clusters of token embeddings—to enhance continuous diffusion language models. Applied to the CoBit architecture, the resulting H-CoBit achieves significant empirical gains, improving MAUVE scores and achieving lower generative perplexity on LM1B and OWT, while also outperforming prior continuous diffusion models on GSM8K. The approach generalizes to other continuous generative paradigms, as shown by consistent improvements when applied to the flow matching model FLM.

By Mathias Ollu, Nikos Komodakis
llmsragdiffusionbenchmarks
More like this →
arXiv Machine Learning
Oct 12

On the Design Space of Continuous Diffusion Language Models

The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.

By Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis B\'ethune, Pierre Ablin, Marco Cuturi
llmsragdiffusion
More like this →