arXiv AI
Jun 11
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
arXiv:2606. 12071v1 Announce Type: cross Abstract: LLMs are increasingly used to generate and judge scientific ideas.
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Related stories
arXiv Computation and Language
Aug 27
Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
The paper introduces Think‑Probe‑Respond (TPR), a lightweight method to improve large language models’ ability to judge the novelty of research ideas. It identifies a systematic bias where models tend to label ideas as "medium novel" despite generating human‑like rationales, and shows that probing hidden states during reasoning and conditioning the final response on these probes boosts novelty judgment accuracy by 22.30%. TPR effectively reduces the medium‑novelty bias across strong baseline models.
arXiv AI
Aug 7
An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics
arXiv:2604. 15145v2 Announce Type: replace Abstract: The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task.
arXiv AI
Jul 31
Do Methods Support the Claims? Intra-Paper Verification for Peer Review
arXiv:2607. 26066v1 Announce Type: cross Abstract: The growing volume of scientific submissions has motivated interest in using large language models (LLMs) to assist peer review.
arXiv AI
Sep 12
NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment
NovGauge is a new benchmark designed to diagnose large language models’ ability to assess scientific paper novelty. It contains 619 paper pairs and 50 multi-paper sets, each labeled along three dimensions—task, problem, and method—by experts from ICLR reviewer overlap claims and survey co-citations. The study evaluates 18 LLMs, revealing high hallucination rates and weak evidence grounding, with the best model achieving only 43‑72% verified F1 across dimensions.
arXiv AI
Jul 2
Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty
arXiv:2606. 25198v2 Announce Type: replace Abstract: Autonomous AI Research promises to accelerate the scientific progress of machine learning.
arXiv AI
4d ago
Large Language Models Extend Known Research But Do Not Create New Research Directions
arXiv:2610.17439v1 Announce Type: new Abstract: The success of autonomous research systems relies on the quality of the scientific hypotheses they generate. Recent studies report that human reviewers...
arXiv Computation and Language
Oct 12
Evaluating LLM Creativity through Long-Tail Performance
The paper proposes that creativity in large language models can be measured by performance on the long tail of a reference input-output distribution, using n‑gram absence from pretraining data as a novelty proxy. Across 15 benchmarks and 26 open‑weight models, the authors find that creativity benchmarks occupy further tail regions than standard benchmarks and that higher‑tail subsets better align with human creativity judgments. They introduce a new diagnostic, q@Tail(α), to assess mean task quality on the tail fraction of existing benchmarks, revealing significant potential for improving LLM creativity through stronger tail performance.
arXiv AI
Jul 2
Measuring the Gap Between Human and LLM Research Ideas
arXiv:2607. 01233v1 Announce Type: cross Abstract: LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference.
arXiv Computation and Language
3d ago
Training Reasoning Models to Generate Diverse Scientific Ideas
The paper introduces a method for training large language models to generate diverse, high‑quality scientific ideas grounded in input documents. By using a set‑level reinforcement learning objective that rewards ideas for being distinct from earlier ones, the authors train an 8B model that outperforms Claude Opus 4.8 and GPT‑6 Astra on a contamination‑free benchmark. Human experts rate the model’s idea sets as equally promising while containing slightly more unique ideas, and automated experiments show comparable executability.
Hugging Face Trending Papers
Jun 10
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks.
arXiv AI
Jun 11
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
arXiv:2606. 11762v1 Announce Type: cross Abstract: Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential.