arXiv AI
BCT: Bayes-Consistent Tempering for Flow Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
1d ago
Loopholing All the Way: Language Modeling with pure Self-Conditioning
The paper introduces LAW (Loopholing All the Way), a continuous generative model that relies entirely on Self‑Conditioning (SC) to propagate expected embeddings across diffusion steps, decoding tokens only once at the end. LAW outperforms both discrete and continuous diffusion models on Sudoku, Maze, and TinyGSM benchmarks, achieving higher accuracy and solving a larger proportion of GSM8K problems. Fine‑tuning a pre‑trained Uniform Diffusion Model with LAW significantly boosts its performance, and on OpenWebText LAW approaches autoregressive models while maintaining a favorable Pareto frontier.
arXiv AI
Jul 28
UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
arXiv:2607. 24507v1 Announce Type: cross Abstract: Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling.
arXiv AI
Oct 5
Large Language Continuous Diffusion Models
The paper introduces Sigma, a large-scale continuous diffusion language model (3B/8B parameters) that uses steerable, low-dimensional ODE/SDE latent trajectories to address the non-smoothness of discrete diffusion models. Sigma is trained blockwise via likelihood optimization, jointly denoises Gaussian-corrupted token embeddings, and learns an optimal embedding geometry, leveraging pre-trained autoregressive weights for faster training. During inference, classifier-free guidance and score temperature are identified as essential for high-fidelity reasoning and coding, and Sigma matches or exceeds discrete models on benchmarks such as GSM8K, Minerva, HumanEval, MBPP, MATH-500, and AIME, while also revealing unique structural benefits like embedding-space steering and graceful degradation for low NFEs.
arXiv Machine Learning
Oct 16
On-Policy Residual Learning for Diffusion Language Models
On-Policy Residual Learning for Diffusion Language Models (ORL) introduces a framework that learns residual context directly from model-generated trajectories, using an entropy-contraction objective to sharpen predictions. The approach co-designs algorithmic and system components—such as asynchronous residual capture, block-diffusion attention, and packed compute-balanced training—to enable practical scaling. Experiments on a 4B DLM converted from Qwen3-4B show up to 6.2× faster tokens per forward and 6.9× higher end-to-end throughput, while also improving speed-quality trade-offs over autoregressive models and transferring effectively to other DLMs like LLaDA-2.1-mini and DiffusionGemma.
arXiv Machine Learning
Jul 7
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
arXiv:2607. 03809v1 Announce Type: new Abstract: Normalising flows provide a powerful variational family for approximate inference, yet individual architectures often fail to generalise across heterogeneous posterior geometries.
arXiv Machine Learning
Jun 10
Few-step Generative Models as Lossy Compression
arXiv:2606. 10450v1 Announce Type: cross Abstract: DiffC provides a principled way to reuse pre-trained diffusion models for lossy compression, but its encoding and decoding procedures remain slow because they require many discretized forward and reverse steps.