The paper argues that traditional semantic similarity fails to capture the true diversity of language models. It introduces a new metric—generative‑process diversity—measured via Normalised Compression Distance on raw outputs, which reveals hidden population structure among 38 models. This metric predicts lower correlated failures across ten benchmark families, independent of semantic similarity or model capability.
arXiv:2609.16454v1 Announce Type: new
Abstract: Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend...
By Kirill Skobelev, Eric Fithian, X. Y. Han
The paper investigates whether large language models (LLMs) capture the full diversity of outputs present in their training data. Using an information‑theoretic approach, the authors compare the conditional entropy of model‑generated outputs with that of the training data, finding that LLMs consistently produce outputs with lower conditional entropy across various models, scales, and decoding strategies. They also extend the analysis to image and text‑conditioned generators, propose a post‑hoc correction method based on matrix‑entropy projection to increase conditional diversity, and provide theoretical guarantees and an efficient algorithm for this correction.
By Youqi Wu, Farzan Farnia
The paper argues that large language models cannot achieve perfect reliability for any task, even with unlimited scale. It establishes that each generative task has an inherent reliability ceiling set by how much output uncertainty can be resolved from observable context, with a resolvable part that can be improved by more context and a subjective part tied to task ambiguity. The authors derive a scaling law showing that performance is limited by the scarcer resource—either training data or model capacity—and explain how this law explains phenomena such as retrieval augmentation and catastrophic forgetting.
By Subhabrata Majumdar
arXiv:2607. 14112v1 Announce Type: cross Abstract: Large language models (LLMs) are evaluated as though perfect reliability is achievable for any task given sufficient scale.
By Subhabrata Majumdar
The paper argues that measuring diversity in AI-generated content using a single scalar score is inherently ambiguous and often misleading. It reviews existing diversity metrics, demonstrates their limitations through axiomatic and empirical analyses, and introduces diversity profiles—curve-valued, condition-aware summaries that evaluate diversity across a range of thresholds, scales, exponents, or orders. These profiles reveal whether comparisons are robust across resolutions or depend on arbitrary parameter choices, offering a more transparent framework for generative AI evaluation.
By Xiuyuan Hu, Xuege Hou, Guoqing Liu, Yang Zhao, Jieran Li, Dongbiao Sun, Jos\'e Miguel Hern\'andez-Lobato, Hao Zhang, Xue Liu
arXiv:2606. 01811v1 Announce Type: cross Abstract: Measuring the diversity of creative outputs is central to evaluating post-training mode collapse, comparing decoding strategies, and quantifying creative behavior in both AI and human writing.
By Matthew Khoriaty, David Williams-King, Shi Feng
arXiv:2506.17871v4 Announce Type: replace-cross
Abstract: Despite their impressive capabilities, aligned large language models (LLMs) often generate outputs that lack diversity. What drives this cons...
By Chenghao Yang, Sida Li, Ari Holtzman
The paper proposes a depth‑aware FrankenMerging technique that blends lower layers of an aligned large language model (LLM) with higher layers of its base counterpart to balance diversity and quality without extra training. Experiments demonstrate that adjusting the layer split controls both lexical and semantic diversity, often surpassing the aligned model in semantic diversity and outperforming temperature sampling in most settings. The method is validated by an ablation that reverses the merge order, supporting the hypothesis about layer‑specific diversity contributions.
By Yasuaki Sumita, Tomoharu Iwata, Toshiyuki Tanaka
arXiv:2505. 20161v2 Announce Type: replace-cross Abstract: Effective generalization in language models depends critically on the diversity of their training data.
By Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostafa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi
arXiv:2512. 15011v3 Announce Type: replace Abstract: Artificial intelligence (AI) increasingly generates the very content used to train future AI systems.
By Damian Hodel, Jevin D. West
The paper critiques generative perplexity (gen‑PPL) as an unreliable metric for evaluating unconditional text generation, showing that it can be minimized by trivial, non‑diverse outputs. The authors demonstrate this by constructing simple samplers that achieve low gen‑PPL yet produce incoherent text, and they propose alternative evaluation tools—including distributional discrepancies, diversity diagnostics, and a compression‑based KL divergence measure—to better assess generative language models. They argue that future progress in diffusion language models should rely on a suite of metrics that capture diverse aspects of generated text quality.
By Antonio Franca, Alexander Tong