Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics
Read the original on arXiv Computation and Language →The paper critiques generative perplexity (gen‑PPL) as an unreliable metric for evaluating unconditional text generation, showing that it can be minimized by trivial, non‑diverse outputs. The authors demonstrate this by constructing simple samplers that achieve low gen‑PPL yet produce incoherent text, and they propose alternative evaluation tools—including distributional discrepancies, diversity diagnostics, and a compression‑based KL divergence measure—to better assess generative language models. They argue that future progress in diffusion language models should rely on a suite of metrics that capture diverse aspects of generated text quality.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.