The paper critiques generative perplexity (gen‑PPL) as an unreliable metric for evaluating unconditional text generation, showing that it can be minimized by trivial, non‑diverse outputs. The authors demonstrate this by constructing simple samplers that achieve low gen‑PPL yet produce incoherent text, and they propose alternative evaluation tools—including distributional discrepancies, diversity diagnostics, and a compression‑based KL divergence measure—to better assess generative language models. They argue that future progress in diffusion language models should rely on a suite of metrics that capture diverse aspects of generated text quality.
arXiv:2606. 08417v1 Announce Type: cross Abstract: Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.
arXiv:2603.02760v2 Announce Type: replace-cross Abstract: Diffusion large language models (dLLMs) have recently attracted significant attention for their ability to enhance diversity, controllability...
arXiv:2605.27268v2 Announce Type: replace-cross Abstract: Modern Large Language Models (LLMs) are often criticized for producing repetitive and homogeneous text, despite possessing vast latent vocabu...
arXiv:2607. 05679v1 Announce Type: cross Abstract: Language models (LMs) exhibit problematic biases, such as stereotypes.
arXiv:2610.29602v1 Announce Type: cross Abstract: Diffusion language models generate by iteratively denoising a masked sequence under bidirectional attention, without the strict left-to-right commitm...
arXiv:2610.19166v1 Announce Type: new Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that co...
The paper offers a unified probabilistic framework for large language models, describing them as probability measures over token sequences defined by autoregressive conditional distributions. Training is cast as maximum‑likelihood estimation solved via stochastic gradient methods, while generation is treated as sequential simulation of the resulting stochastic process. It also explores how the asymmetry of the Kullback–Leibler divergence relates to hallucination and the distinction between plausibility and truth, and extends the perspective to diffusion models that generate data by simulating a reverse‑time stochastic process.
The paper introduces Seq-Drifting, a new framework that enables diffusion language models to generate an entire text sequence in a single forward pass. By applying a drift field to continuous token representations, the model learns a direct mapping from Gaussian noise to text, using attraction to guide outputs toward contextually plausible tokens and repulsion to prevent repetition and collapse. Experiments show that Seq-Drifting achieves a perplexity of 36.01 on LM1B and performs competitively in unconditional, conditional, translation, summarization, and reasoning tasks.
arXiv:2510. 21891v2 Announce Type: replace-cross Abstract: To deploy large language models (LLMs) in high-stakes application domains that require substantively accurate responses to open-ended prompts, we need reliable, computationally inexpensive methods that assess the trustworthiness of long-form responses generated by LLMs.
The paper introduces DLM-One, a score‑distillation framework that enables one‑step sequence generation with continuous diffusion language models (DLMs). By aligning a student model’s outputs with a pretrained teacher DLM’s score function in the forward‑diffused noisy space, DLM-One removes the need for iterative refinement. Experiments across various DLM architectures show up to ~2000× speedup in sampling steps and ~500× in wall‑clock time while retaining competitive performance, and the authors propose an adversarially‑regularized two‑stage training scheme to mitigate student degeneration.
arXiv:2609.40235v1 Announce Type: cross Abstract: Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (...
arXiv:2503. 13505v3 Announce Type: replace-cross Abstract: Generative Pretrained Transformers (GPTs) are foundational Large Language Models (LLMs) for text generation.