← Back to all news
arXiv Machine Learning By Andrea Miele, Ilija Bogunovic

Spherical Brownian Diffusion for Language Modeling

Read the original on arXiv Machine Learning →

The paper introduces Spherical Brownian Diffusion, a new continuous diffusion language model that uses Brownian motion on the unit sphere as its forward process. It provides a principled heat semigroup, an additive dimension‑free noise clock, and an exact ancestral bridge, while a clean‑token posterior learned with cross‑entropy enables multiple samplers. Experiments on Sudoku and GSM8K show that the choice of corruption process has little effect, whereas sampler selection can significantly impact performance, with memory and token‑selection effects differing across tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 28

UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

arXiv:2607. 24507v1 Announce Type: cross Abstract: Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling.

By Xiaoyi Jiang, Jingyuan Li, Yixuan Jiang, Wei Liu, Yi Zhu, Zuoqiang Shi, Pipi Hu
llmsdiffusion
More like this →
arXiv Machine Learning
Oct 12

On the Design Space of Continuous Diffusion Language Models

The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.

By Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis B\'ethune, Pierre Ablin, Marco Cuturi
llmsragdiffusion
More like this →
arXiv Machine Learning
Jun 2

Consistent Diffusion Language Models

arXiv:2605. 00161v2 Announce Type: replace Abstract: Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps.

By Hasan Amin, Yuan Gao, Yaser Souri, Subhojit Som, Ming Yin, Rajiv Khanna, Xia Song
llmsdiffusionefficiency
More like this →
arXiv Computation and Language
Aug 28

Forward-Free Diffusion Language Models with BPTT-Free Looped Refinement

Forward-Free Diffusion Language Models with BPTT-Free Looped Refinement (FReDA) removes the need for a hand‑designed forward process in diffusion language modeling by treating model‑generated drafts as implicit intermediate states and refining them recursively. The approach detaches earlier refinement passes, backpropagating only through the final pass, and supports both self‑refinement and Best‑of‑N candidate selection. In sub‑8B experiments, FReDA‑4B surpasses larger diffusion baselines on reasoning and coding tasks, achieving up to 15% absolute gains and a 1.5‑1.8× speedup while scaling well with additional refinement steps.

By Haotian Sun, Rushi Qiang, Yuqian Zheng, Bo Dai
llmsdiffusionbenchmarks
More like this →