arXiv Computation and Language

GENIE: A Fine-Grained Measure for Novelty

The paper introduces GENIE, a fine‑grained metric designed to assess the novelty of responses generated by large language models on a task‑specific basis. Unlike holistic metrics, GENIE evaluates novelty across multiple features, revealing which aspects of model outputs are truly novel. The authors also apply GENIE to evaluate mitigation methods aimed at enhancing creativity, showing where such methods succeed or fall short.

arXiv Computation and Language

Oct 12

Evaluating LLM Creativity through Long-Tail Performance

The paper proposes that creativity in large language models can be measured by performance on the long tail of a reference input-output distribution, using n‑gram absence from pretraining data as a novelty proxy. Across 15 benchmarks and 26 open‑weight models, the authors find that creativity benchmarks occupy further tail regions than standard benchmarks and that higher‑tail subsets better align with human creativity judgments. They introduce a new diagnostic, q@Tail(α), to assess mean task quality on the tail fraction of existing benchmarks, revealing significant potential for improving LLM creativity through stronger tail performance.

By Yichen Wang, Tenghao Huang, Vishakh Padmakumar, Mina Lee

llms benchmarks

Hugging Face Trending Papers

Jun 10

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks.

llms rag agents benchmarks

arXiv AI

Jun 11

Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

arXiv:2606. 11762v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential.

By Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Mohor Banerjee, Swaagat Bikash Saikia, Alvin Chan

llms rag agents benchmarks

arXiv AI

Jul 3

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity

arXiv:2510. 20091v3 Announce Type: replace-cross Abstract: Creativity is often seen as a hallmark of human intelligence.

By Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li

llms benchmarks

arXiv Computation and Language

Aug 27

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

The paper introduces Think‑Probe‑Respond (TPR), a lightweight method to improve large language models’ ability to judge the novelty of research ideas. It identifies a systematic bias where models tend to label ideas as "medium novel" despite generating human‑like rationales, and shows that probing hidden states during reasoning and conditioning the final response on these probes boosts novelty judgment accuracy by 22.30%. TPR effectively reduces the medium‑novelty bias across strong baseline models.

By Tim Schopf, Tobias Schreieder, Akiko Aizawa

llms safety

arXiv Computation and Language

Oct 15

Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

arXiv:2610.02022v2 Announce Type: replace Abstract: AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models....

By Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope

llms

arXiv Computation and Language

Sep 15

Measuring the Creativity of Frontier LLMs in Automated Research

arXiv:2609.14057v1 Announce Type: new Abstract: Frontier LLMs are increasingly capable of conducting automated research, yet their creativity in this setting has not been systematically evaluated. In...

By Yiheng Zhao, Mengzhuo Chen, Chengming Hu, Pengyi Liao, Yiran Pang

llms

arXiv AI

Aug 7

An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics

arXiv:2604. 15145v2 Announce Type: replace Abstract: The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task.

By Miri Liu, ChengXiang Zhai

llms rag benchmarks

arXiv AI

Aug 21

Are LLMs becoming similarly creative? Evidence from three years of models

arXiv:2608. 19437v1 Announce Type: cross Abstract: Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality.

By Nirav Patel, Josiah Crossman, Eva Aggarwal, Emily Wenger

llms rag benchmarks

arXiv Machine Learning

Aug 4

LLM generation novelty through the lens of semantic similarity

arXiv:2510. 27313v3 Announce Type: replace Abstract: Generation novelty is a key indicator of an LLM's ability to generalize, yet measuring it against full pretraining corpora is computationally challenging.

By Philipp Davydov, Ameya Prabhu, Matthias Bethge, Elisa Nguyen, Seong Joon Oh

llms rag fine-tuning safety

arXiv AI

Sep 1

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

arXiv:2608.30047v1 Announce Type: new Abstract: Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they...

By Shitanshu Bhushan, Yunxiang Zhang, Lu Wang

llms agents

arXiv AI

Jul 2

Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty

arXiv:2606. 25198v2 Announce Type: replace Abstract: Autonomous AI Research promises to accelerate the scientific progress of machine learning.

By Antonis Antoniades, Deepak Nathani, Ritam Saha, Alfonso Amayuelas, Ivan Bercovich, Zhaotian Weng, Vignesh Baskaran, Kunal Bhatia, William Yang Wang

llms agents reinforcement-learning

Related stories