arXiv AI August 26, 2026 By Alessandro Tutone, Giorgio Franceschelli, Mirco Musolesi

The Limits of Automatic Evaluation of Creativity in Large Language Models

The paper examines whether existing automatic methods can reliably assess creativity in text produced by large language models (LLMs). By collecting human ratings on 11 creativity dimensions for both human and AI short stories, the authors compare these judgments with automated metrics and LLM-as-a-Judge evaluations. The results show a significant misalignment: automated metrics and LLM judges favor AI-generated stories and show near-zero correlation with human assessments, revealing fundamental limitations in current computational approaches to evaluating creative text.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 3

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity

arXiv:2510. 20091v3 Announce Type: replace-cross Abstract: Creativity is often seen as a hallmark of human intelligence.

By Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li

llms benchmarks

arXiv AI

Jun 11

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

arXiv:2606. 11762v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential.

By Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Mohor Banerjee, Swaagat Bikash Saikia, Alvin Chan

llms rag agents benchmarks

Hugging Face Trending Papers

Jun 10

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks.

llms rag agents benchmarks

arXiv AI

Jun 17

Do Large Language Models Always Tell The Same Stories?

arXiv:2606. 17350v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have enabled the generation of high-quality prose, yet the question of whether these models are capable of generating diverse outputs remains contested.

By Thennal DK, Hans Ole Hatzel

llms

arXiv AI

Aug 21

Are LLMs becoming similarly creative? Evidence from three years of models

arXiv:2608. 19437v1 Announce Type: cross Abstract: Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality.

By Nirav Patel, Josiah Crossman, Eva Aggarwal, Emily Wenger

llms rag benchmarks

arXiv AI

Aug 10

CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

arXiv:2608. 07460v1 Announce Type: cross Abstract: While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.

By Ananya Sahu, Mohit Bansal, Elias Stengel-Eskin

llms reinforcement-learning safety

One email a morning, machine-written

Related stories

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Do Large Language Models Always Tell The Same Stories?

Are LLMs becoming similarly creative? Evidence from three years of models

CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity