arXiv AI
Jul 3
CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
arXiv:2510. 20091v3 Announce Type: replace-cross Abstract: Creativity is often seen as a hallmark of human intelligence.
The Limits of Automatic Evaluation of Creativity in Large Language Models
The paper examines whether existing automatic methods can reliably assess creativity in text produced by large language models (LLMs). By collecting human ratings on 11 creativity dimensions for both human and AI short stories, the authors compare these judgments with automated metrics and LLM-as-a-Judge evaluations. The results show a significant misalignment: automated metrics and LLM judges favor AI-generated stories and show near-zero correlation with human assessments, revealing fundamental limitations in current computational approaches to evaluating creative text.
Related stories
arXiv Computation and Language
Sep 1
CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text
arXiv:2608.30754v1 Announce Type: new Abstract: Evaluating creativity in large language model (LLM) outputs remains challenging because creativity is multidimensional and human-centered. We examine h...
Hugging Face Trending Papers
Aug 31
CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text
Evaluating creativity in large language model (LLM) outputs remains challenging because creativity is multidimensional and human-centered. We examine how reliably LLMs evaluate short literary text in...
arXiv AI
Jun 11
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
arXiv:2606. 11762v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential.
Hugging Face Trending Papers
Jun 10
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks.
arXiv AI
Jun 17
Do Large Language Models Always Tell The Same Stories?
arXiv:2606. 17350v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have enabled the generation of high-quality prose, yet the question of whether these models are capable of generating diverse outputs remains contested.
arXiv AI
Aug 21
Are LLMs becoming similarly creative? Evidence from three years of models
arXiv:2608. 19437v1 Announce Type: cross Abstract: Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality.
arXiv Computation and Language
Sep 2
Evaluating Style-Personalized Text Generation: Challenges and Directions
The paper "Evaluating Style-Personalized Text Generation: Challenges and Directions" examines the difficulties of assessing text that is tailored to individual users’ styles. It critiques common metrics such as BLEU, embeddings, and LLM-as-judges, and introduces a style discrimination benchmark covering domain discrimination, authorship attribution, and LLM-generated personalized versus non-personalized discrimination across eight writing tasks. The study finds that ensembles of diverse evaluation metrics outperform single-evaluator approaches and offers guidance for reliable assessment of style-personalized generation.
arXiv AI
Aug 10
CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity
arXiv:2608. 07460v1 Announce Type: cross Abstract: While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.
arXiv Computation and Language
2d ago
Evaluating LLM Creativity through Long-Tail Performance
The paper proposes that creativity in large language models can be measured by performance on the long tail of a reference input-output distribution, using n‑gram absence from pretraining data as a novelty proxy. Across 15 benchmarks and 26 open‑weight models, the authors find that creativity benchmarks occupy further tail regions than standard benchmarks and that higher‑tail subsets better align with human creativity judgments. They introduce a new diagnostic, q@Tail(α), to assess mean task quality on the tail fraction of existing benchmarks, revealing significant potential for improving LLM creativity through stronger tail performance.
arXiv AI
Aug 10
Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models
arXiv:2608. 07243v1 Announce Type: new Abstract: Generative models are often evaluated through singular artifacts, whereas human creativity typically emerges through iterative generation, appraisal, and refinement.
Hugging Face Trending Papers
Aug 24
LITERARYBIGFIVE: Author-Personalized Text Generation in a Unified Interpretable Space
Personalized text generation for authors and literary writing is essential for applications such as adaptive writing assistants, creative support tools, and computational literary analysis. However, e...