arXiv:2608.23551v1 Announce Type: cross Abstract: Recent advances in continuous diffusion and flow-based language models (LMs) have achieved performance competitive with discrete LMs. However, existi...
arXiv:2610.28559v1 Announce Type: cross Abstract: Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet ef...
arXiv:2606. 16847v1 Announce Type: cross Abstract: Diffusion Large Language Models (dLLMs) offer a promising avenue for parallel generation but face a trade-off between decoding speed and quality.
Cola2 is a hierarchical language modeling framework that jointly optimizes continuous semantic compression, reconstruction–generation balance, and continuous–discrete co‑adaptation. It dynamically segments text into semantic spans, controls compression via negative feedback, and trains a block‑causal masked diffusion decoder with clean, noisy, and dropped latent conditions. Experiments on five downstream tasks with 210 billion training tokens show competitive scaling, and an unconverged checkpoint with 346 billion source tokens outperforms Llama‑3‑8B on HellaSwag and StoryCloze.
The paper introduces Dependency-Aware Revocable Decoding (DARD), a training‑free framework for diffusion large language models that separates tokens into masked, candidate, and unmasked states. DARD verifies candidate tokens using a selective context that excludes less reliable tokens and adaptively regulates their influence on subsequent decoding. Experiments on 12 textual and multimodal benchmarks across three open‑source dLLMs show that DARD improves the speed‑quality Pareto frontier, achieving a 2.71× speedup and a 4.35‑point CIDEr gain over Saber on Flickr30K.
The paper introduces Seq-Drifting, a new framework that enables diffusion language models to generate an entire text sequence in a single forward pass. By applying a drift field to continuous token representations, the model learns a direct mapping from Gaussian noise to text, using attraction to guide outputs toward contextually plausible tokens and repulsion to prevent repetition and collapse. Experiments show that Seq-Drifting achieves a perplexity of 36.01 on LM1B and performs competitively in unconditional, conditional, translation, summarization, and reasoning tasks.
arXiv:2606. 08417v1 Announce Type: cross Abstract: Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.
arXiv:2605.27971v2 Announce Type: replace-cross Abstract: When large language models are fine-tuned to generate persona- or tone-conditioned responses, their output diversity is severely limited--a f...
The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.
The paper introduces DCOCO (DFlash Contextual Corrector), a lightweight mechanism that restores intra-block dependencies in speculative decoding by evaluating multiple candidate tokens for contextual compatibility across adjacent draft positions. DCOCO maintains parallel drafting while enabling contextually informed token selection at low additional cost. Across multiple verifier backbones and benchmarks, DCOCO achieves the highest average throughput, delivering speedups of up to 6.6× over vanilla decoding and outperforming competing methods by 6–23% under concurrent serving loads.
The paper investigates how to effectively pre‑train language models when the data budget is limited but compute is plentiful. It shows that increasing model size only improves performance up to an optimal point, after which overfitting degrades generalization, and that this optimal size varies with both the data budget and downstream tasks. To overcome the inefficiencies of standard Transformers in this regime, the authors propose recursive Transformers that reuse a shared block across depth and employ factorized embeddings, achieving better results than standard models on 10M–100M word pre‑training budgets and competitive performance with BabyLM Challenge 2025 winners.
Speculative decoding, in which a lightweight draft model first generates a draft sequence that is then verified in parallel by the target model, has become a prevalent paradigm for accelerating large language model inference. Recent work such as DFlash further boosts drafting efficiency by leveraging diffusion drafters, whose parallel denoising mechanism enables draft generation in a single forward pass.