arXiv AI
2d ago
Rectified Language Flow
arXiv:2610.33811v1 Announce Type: new Abstract: Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of aut...
Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation
Cola2 is a hierarchical language modeling framework that jointly optimizes continuous semantic compression, reconstruction–generation balance, and continuous–discrete co‑adaptation. It dynamically segments text into semantic spans, controls compression via negative feedback, and trains a block‑causal masked diffusion decoder with clean, noisy, and dropped latent conditions. Experiments on five downstream tasks with 210 billion training tokens show competitive scaling, and an unconverged checkpoint with 346 billion source tokens outperforms Llama‑3‑8B on HellaSwag and StoryCloze.
Related stories
arXiv Machine Learning
Oct 12
Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation
The paper proposes Predictive-State Flows (PSF), a new architecture for continuous diffusion language models that separates the diffusion prior’s role to generate high‑level predictive states and the decoder’s role to resolve token‑level choices. PSF partitions sequences into blocks, uses a rectified‑flow prior to produce predictive states, and then a block‑causal decoder generates all blocks in parallel, reducing token‑level sequential depth. Experiments on OpenWebText show that a 197M‑parameter PSF model with eight predictive states achieves a generative perplexity of 22.8 and a MAUVE@512 of 0.93 while using far fewer FLOPs than a comparable ELF‑B model.
arXiv Computation and Language
Sep 14
Representation-based Masked Diffusion Model
The paper introduces Representation-based Masked Diffusion Model (RMDM), a new framework for language modeling that improves upon existing Masked Diffusion Models by incorporating global semantic guidance. RMDM encodes text into a continuous semantic space with a pretrained encoder, normalizes this representation to a Gaussian prior via an invertible transformation, and then trains a masked diffusion model conditioned on this latent representation to coordinate parallel token updates. Experiments show that RMDM yields higher generation quality, especially when using aggressive few‑step sampling.
arXiv Machine Learning
Sep 24
Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders
The paper introduces LLMAE, a technique that transforms a pretrained decoder-only language model into a continuous text autoencoder by inserting a fixed-length latent bottleneck into its internal activations. Using a 270M Gemma 3 model with structured attention masks, LoRA adaptation, and KL regularization, LLMAE achieves near-perfect reconstruction of text sequences up to 1024 tokens. The authors further show that the resulting latent representation can be leveraged to train a latent text diffusion model for detailed image captioning, demonstrating downstream utility.
arXiv AI
Oct 21
Large Flow Language Models
arXiv:2610.28559v1 Announce Type: cross Abstract: Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet ef...
arXiv Computer Vision
Oct 22
BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing limitations of autoregressive models by allowing iterative bidirectional prediction. The method employs a two‑stage training strategy—Decoupled Uni‑ and Cross‑Modal Training followed by supervised fine‑tuning—to establish cross‑modal correspondence and specialize for bidirectional generation. To mitigate train‑test mismatch, Generation‑Aware Self‑Correction exposes the model to its own predictions during training and applies correction passes early in sampling to refine unmasked tokens, achieving competitive results on HumanML3D and KIT‑ML.
arXiv AI
Oct 12
One-Step Text Generation by Seq-Drifting
The paper introduces Seq-Drifting, a new framework that enables diffusion language models to generate an entire text sequence in a single forward pass. By applying a drift field to continuous token representations, the model learns a direct mapping from Gaussian noise to text, using attraction to guide outputs toward contextually plausible tokens and repulsion to prevent repetition and collapse. Experiments show that Seq-Drifting achieves a perplexity of 36.01 on LM1B and performs competitively in unconditional, conditional, translation, summarization, and reasoning tasks.
Hugging Face Trending Papers
Sep 28
BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing the limitations of autoregressive models in capturing bidirectional dependencies between language and motion. The approach employs a two‑stage training strategy—decoupled uni‑ and cross‑modal pretraining followed by supervised fine‑tuning—to establish robust cross‑modal correspondence, and incorporates Generation‑Aware Self‑Correction to mitigate error propagation during inference. Experiments on HumanML3D and KIT‑ML show competitive performance on both text‑to‑motion and motion‑to‑text tasks, demonstrating the effectiveness of the proposed training and correction mechanisms.
arXiv AI
Aug 17
CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing
arXiv:2608. 13925v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) accelerate language generation by predicting multiple masks in a single forward pass.
arXiv AI
Oct 5
Large Language Continuous Diffusion Models
The paper introduces Sigma, a large-scale continuous diffusion language model (3B/8B parameters) that uses steerable, low-dimensional ODE/SDE latent trajectories to address the non-smoothness of discrete diffusion models. Sigma is trained blockwise via likelihood optimization, jointly denoises Gaussian-corrupted token embeddings, and learns an optimal embedding geometry, leveraging pre-trained autoregressive weights for faster training. During inference, classifier-free guidance and score temperature are identified as essential for high-fidelity reasoning and coding, and Sigma matches or exceeds discrete models on benchmarks such as GSM8K, Minerva, HumanEval, MBPP, MATH-500, and AIME, while also revealing unique structural benefits like embedding-space steering and graceful degradation for low NFEs.
arXiv AI
Oct 19
Rethinking User Encoders: Routing Diffusion States for Scenario-Adaptive User Representation
The paper introduces FINDERS, a diffusion‑language‑model framework that replaces static user encoders with a dynamic routing system. FINDERS uses four operations—denoise, factorize, route, and compress—to create scenario‑adaptive user representations by organizing denoising states into transferable cross‑modal semantics and modality‑specific factors. The authors also build BehaviorQA, a large corpus with explicit behavioral rationales, and demonstrate through extensive experiments and online A/B tests that FINDERS outperforms existing methods across ten industrial scenarios.
arXiv AI
Sep 30
On Trajectory-Aware Training for Masked Diffusion Language Models
arXiv:2609.37974v1 Announce Type: cross Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The m...