← Back to all news
arXiv Machine Learning By Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aur\'elien Lucchi, Antonio Orvieto, Caglar Gulcehre, Pascal Frossard

Loopholing All the Way: Language Modeling with pure Self-Conditioning

Read the original on arXiv Machine Learning →

The paper introduces LAW (Loopholing All the Way), a continuous generative model that relies entirely on Self‑Conditioning (SC) to propagate expected embeddings across diffusion steps, decoding tokens only once at the end. LAW outperforms both discrete and continuous diffusion models on Sudoku, Maze, and TinyGSM benchmarks, achieving higher accuracy and solving a larger proportion of GSM8K problems. Fine‑tuning a pre‑trained Uniform Diffusion Model with LAW significantly boosts its performance, and on OpenWebText LAW approaches autoregressive models while maintaining a favorable Pareto frontier.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 12

On the Design Space of Continuous Diffusion Language Models

The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.

By Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis B\'ethune, Pierre Ablin, Marco Cuturi
llmsragdiffusion
More like this →
arXiv Machine Learning
Sep 22

Optimizers for Diffusion Models: A Controlled Benchmark

The paper introduces the first controlled benchmark for optimizers in discrete diffusion models, evaluating seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS‑M, Schedule‑Free) across four diffusion formulations: masked diffusion on text8, uniform diffusion on QM9 and LM1B, and Gaussian diffusion on CelebA‑64. Each optimizer undergoes the same search protocol and is retrained with full budget and multiple seeds, revealing that AdamW, while strong, is not universally optimal and that optimizers validated on autoregressive language models (Muon, MARS‑M, SOAP) can outperform tuned AdamW on certain tasks.

By Arman Bolatov, Egor Shulgin, David Li, Abduragim Shtanchaev, Sebastian U. Stich, Maxim Panov, Eric Moulines, Peter Richt\'arik, Martin Tak\'a\v{c}
llmsdiffusionbenchmarks
More like this →
arXiv AI
Jul 17

Token Time Continuous Diffusion for Language Modeling

arXiv:2607. 14106v1 Announce Type: cross Abstract: In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others.

By Parikshit Bansal, Sujay Sanghavi
llmsdiffusion
More like this →