← Back to all news
arXiv Machine Learning

Loopholing All the Way: Language Modeling with pure Self-Conditioning

The paper introduces LAW (Loopholing All the Way), a continuous generative model that relies entirely on Self‑Conditioning (SC) to propagate expected embeddings across diffusion steps, decoding tokens only once at the end. LAW outperforms both discrete and continuous diffusion models on Sudoku, Maze, and TinyGSM benchmarks, achieving higher accuracy and solving a larger proportion of GSM8K problems. Fine‑tuning a pre‑trained Uniform Diffusion Model with LAW significantly boosts its performance, and on OpenWebText LAW approaches autoregressive models while maintaining a favorable Pareto frontier.

Related stories

arXiv Machine Learning
Oct 12

On the Design Space of Continuous Diffusion Language Models

The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.

By Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis B\'ethune, Pierre Ablin, Marco Cuturi
llmsragdiffusion
More like this →
arXiv Machine Learning
Sep 22

Optimizers for Diffusion Models: A Controlled Benchmark

The paper introduces the first controlled benchmark for optimizers in discrete diffusion models, evaluating seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS‑M, Schedule‑Free) across four diffusion formulations: masked diffusion on text8, uniform diffusion on QM9 and LM1B, and Gaussian diffusion on CelebA‑64. Each optimizer undergoes the same search protocol and is retrained with full budget and multiple seeds, revealing that AdamW, while strong, is not universally optimal and that optimizers validated on autoregressive language models (Muon, MARS‑M, SOAP) can outperform tuned AdamW on certain tasks.

By Arman Bolatov, Egor Shulgin, David Li, Abduragim Shtanchaev, Sebastian U. Stich, Maxim Panov, Eric Moulines, Peter Richt\'arik, Martin Tak\'a\v{c}
llmsdiffusionbenchmarks
More like this →
arXiv AI
Jul 17

Token Time Continuous Diffusion for Language Modeling

arXiv:2607. 14106v1 Announce Type: cross Abstract: In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others.

By Parikshit Bansal, Sujay Sanghavi
llmsdiffusion
More like this →
arXiv Machine Learning
Jun 2

Consistent Diffusion Language Models

arXiv:2605. 00161v2 Announce Type: replace Abstract: Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps.

By Hasan Amin, Yuan Gao, Yaser Souri, Subhojit Som, Ming Yin, Rajiv Khanna, Xia Song
llmsdiffusionefficiency
More like this →
arXiv Computation and Language
1d ago

Hierarchical Masked Diffusion Language Models

arXiv:2610.29255v1 Announce Type: new Abstract: Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked...

By Parul Gupta, Satwik, Avni Aggarwal, Bhavesh Gurnani, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla
llmsdiffusion
More like this →
arXiv AI
Oct 2

Hierarchical Continuous Diffusion Language Models

Hierarchical Continuous Diffusion Language Models (HC‑DLM) combine discrete token generation with a continuous latent trajectory in a single denoising process, addressing the independence bottleneck of parallel decoding in discrete diffusion models. The training objective is derived from a variational bound on token likelihood, and the latent state is the sole persistent generative element, with tokens read out and fed back at each step. Experiments on Sudoku, Countdown, and LM1B show HC‑DLM outperforming both discrete and continuous diffusion baselines in puzzle accuracy and generative perplexity.

By Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing
llmsdiffusion
More like this →