arXiv Machine Learning
Jun 2
BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
arXiv:2606. 02241v1 Announce Type: new Abstract: Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion?
Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning
The paper introduces Mixture‑Likelihood Fine‑Tuning (Mix‑FT), a method that fine‑tunes a pretrained masked diffusion language model on an importance‑weighted bound of its own sampler’s likelihood, without altering the architecture or requiring a teacher. Experiments on LLaDA‑8B and a 110M‑parameter block diffusion model show that Mix‑FT outperforms continued cross‑entropy, Alpha diffusion, and SDTT distillation across various step counts, yielding higher‑quality continuations and better scores from multiple human judges.
Related stories
arXiv AI
Oct 1
Distilling Diffusion Score Discrepancy for Efficient Training Data Attribution
The paper introduces a new method for training data attribution in diffusion models called TID, which uses a local score discrepancy measure and can be estimated without retraining. It further distills this approach into TIDE, a forward‑only student that reproduces the teacher’s rankings using internal activations, achieving comparable accuracy at dramatically lower query cost. Experiments on CIFAR‑10, ArtBench‑10, and MS‑COCO show that TID outperforms existing methods and TIDE attributes samples in milliseconds, faster than generation itself.
arXiv AI
Oct 15
Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models
arXiv:2610.17590v1 Announce Type: cross Abstract: Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to t...
arXiv Machine Learning
Sep 22
Optimizers for Diffusion Models: A Controlled Benchmark
The paper introduces the first controlled benchmark for optimizers in discrete diffusion models, evaluating seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS‑M, Schedule‑Free) across four diffusion formulations: masked diffusion on text8, uniform diffusion on QM9 and LM1B, and Gaussian diffusion on CelebA‑64. Each optimizer undergoes the same search protocol and is retrained with full budget and multiple seeds, revealing that AdamW, while strong, is not universally optimal and that optimizers validated on autoregressive language models (Muon, MARS‑M, SOAP) can outperform tuned AdamW on certain tasks.
arXiv Machine Learning
Jun 15
Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens
arXiv:2606. 14620v1 Announce Type: new Abstract: Open diffusion language models are marketed as parallel, non-autoregressive decoders, yet the order in which a shipped checkpoint actually commits its tokens is almost never measured.
Hugging Face Trending Papers
Oct 12
PIG: Parallel Inconsistency Guidance for Accelerating Diffusion Language Models
Diffusion language models accelerate generation by decoding multiple masked positions per forward pass. However, accuracy is known to drop when parallel decoding trajectories deviate from their sequen...
arXiv Machine Learning
Oct 5
Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs
The paper compares two low‑budget methods for converting a 30B Mixture‑of‑Experts autoregressive language model into a diffusion language model. One method updates a subset of the model’s weights in‑place, while the other freezes the context tower and conditions on a frozen causal copy via cross‑attention. With only 1B training tokens, the frozen‑tower approach achieves a HumanEval pass@10 score of 71.60 versus 6.19 for the in‑place method, and retains 95% of the parent’s GSM8K and 99% of its MMLU‑Pro performance.
arXiv AI
Jun 10
Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
arXiv:2606. 10829v1 Announce Type: cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled.
arXiv AI
Sep 30
On Trajectory-Aware Training for Masked Diffusion Language Models
arXiv:2609.37974v1 Announce Type: cross Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The m...
arXiv Machine Learning
Oct 22
Looped Uniform Diffusion Language Models
The paper introduces looped uniform diffusion language models (UDLMs), which iteratively denoise token sequences and improve training efficiency by applying a few early Transformer layers repeatedly within each denoising step. Experiments show that looped UDLMs achieve the same validation negative log‑likelihood as a non‑looped model trained on roughly half the tokens, while reducing training FLOPs by up to 2.8× on OpenWebText and 3.3× on FineWeb‑Edu. Additionally, with a matched number of denoising steps, looped UDLMs attain up to 9.8% lower generation perplexity, and sampling smaller loop counts more often during training further boosts inference efficiency without sacrificing accuracy.
arXiv AI
Jun 3
Fast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference
arXiv:2606. 02955v1 Announce Type: cross Abstract: Diffusion large language models promise parallel token generation, yet inference remains bottlenecked by deciding which masked tokens can be safely committed together.
arXiv Computation and Language
Oct 22
Hierarchical Masked Diffusion Language Models
arXiv:2610.29255v1 Announce Type: new Abstract: Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked...