arXiv:2610.33811v1 Announce Type: new
Abstract: Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of aut...
By Baiyu Su, Runlong Liao, Lizhang Chen, Qiang Liu
The paper proposes Predictive-State Flows (PSF), a new architecture for continuous diffusion language models that separates the diffusion prior’s role to generate high‑level predictive states and the decoder’s role to resolve token‑level choices. PSF partitions sequences into blocks, uses a rectified‑flow prior to produce predictive states, and then a block‑causal decoder generates all blocks in parallel, reducing token‑level sequential depth. Experiments on OpenWebText show that a 197M‑parameter PSF model with eight predictive states achieves a generative perplexity of 22.8 and a MAUVE@512 of 0.93 while using far fewer FLOPs than a comparable ELF‑B model.
By Jinjin He, Xinsong Feng, Lucheng Fu, Sinan Wang, Bo Zhu, Jiayun Wang
The paper introduces Representation-based Masked Diffusion Model (RMDM), a new framework for language modeling that improves upon existing Masked Diffusion Models by incorporating global semantic guidance. RMDM encodes text into a continuous semantic space with a pretrained encoder, normalizes this representation to a Gaussian prior via an invertible transformation, and then trains a masked diffusion model conditioned on this latent representation to coordinate parallel token updates. Experiments show that RMDM yields higher generation quality, especially when using aggressive few‑step sampling.
By Yangrong Hu, Ding Huang, Xueyu Zhou, Jian Huang
The paper introduces LLMAE, a technique that transforms a pretrained decoder-only language model into a continuous text autoencoder by inserting a fixed-length latent bottleneck into its internal activations. Using a 270M Gemma 3 model with structured attention masks, LoRA adaptation, and KL regularization, LLMAE achieves near-perfect reconstruction of text sequences up to 1024 tokens. The authors further show that the resulting latent representation can be leveraged to train a latent text diffusion model for detailed image captioning, demonstrating downstream utility.
By Arkanath Pathak, Unnat Jain, Alexander C. Berg
arXiv:2610.28559v1 Announce Type: cross
Abstract: Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet ef...
By Arnav Chavan, Chanhyuk Lee, Udbhav Bamba, Jaehoon Yoo, Seunghoon Hong, Aditi Raghunathan, Jinwoo Kim
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing limitations of autoregressive models by allowing iterative bidirectional prediction. The method employs a two‑stage training strategy—Decoupled Uni‑ and Cross‑Modal Training followed by supervised fine‑tuning—to establish cross‑modal correspondence and specialize for bidirectional generation. To mitigate train‑test mismatch, Generation‑Aware Self‑Correction exposes the model to its own predictions during training and applies correction passes early in sampling to refine unmasked tokens, achieving competitive results on HumanML3D and KIT‑ML.
By Wanjiang Weng, Yongliang Wu, Xiaofeng Tan, Xingyu Zhu, Wenbo Zhu, Hongsong Wang