arXiv Computation and Language August 27, 2026 By Tim Schopf, Tobias Schreieder, Akiko Aizawa

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

Read the original on arXiv Computation and Language →

The paper introduces Think‑Probe‑Respond (TPR), a lightweight method to improve large language models’ ability to judge the novelty of research ideas. It identifies a systematic bias where models tend to label ideas as "medium novel" despite generating human‑like rationales, and shows that probing hidden states during reasoning and conditioning the final response on these probes boosts novelty judgment accuracy by 22.30%. TPR effectively reduces the medium‑novelty bias across strong baseline models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jun 11

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

arXiv:2606. 12071v1 Announce Type: cross Abstract: LLMs are increasingly used to generate and judge scientific ideas.

By Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria

llms benchmarks

arXiv AI

Sep 12

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

NovGauge is a new benchmark designed to diagnose large language models’ ability to assess scientific paper novelty. It contains 619 paper pairs and 50 multi-paper sets, each labeled along three dimensions—task, problem, and method—by experts from ICLR reviewer overlap claims and survey co-citations. The study evaluates 18 LLMs, revealing high hallucination rates and weak evidence grounding, with the best model achieving only 43‑72% verified F1 across dimensions.

By Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang

llms benchmarks safety

arXiv AI

Aug 7

An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics

arXiv:2604. 15145v2 Announce Type: replace Abstract: The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task.

By Miri Liu, ChengXiang Zhai

llms rag benchmarks

arXiv AI

Jul 31

Do Methods Support the Claims? Intra-Paper Verification for Peer Review

arXiv:2607. 26066v1 Announce Type: cross Abstract: The growing volume of scientific submissions has motivated interest in using large language models (LLMs) to assist peer review.

By Ranjitha Shivaprasad Ballakuraya, Arash Mahyari, Ashok Srinivasan

llms safety

arXiv AI

Jun 9

Contemporary AI lacks the imagination to diverge or negate in science

arXiv:2606. 08251v1 Announce Type: cross Abstract: Bold projections that artificial intelligence will accelerate scientific discovery have raced ahead of evidence from working scientists, and the field still lacks large-scale, scientist-in-the-loop tests of these claims.

By Honglin Bao, Siyang Wu, Xiao Liu, Sida Li, Shiyun Cao, James A. Evans

llms reinforcement-learning benchmarks

arXiv Computation and Language

Sep 15

Measuring the Creativity of Frontier LLMs in Automated Research

arXiv:2609.14057v1 Announce Type: new Abstract: Frontier LLMs are increasingly capable of conducting automated research, yet their creativity in this setting has not been systematically evaluated. In...

By Yiheng Zhao, Mengzhuo Chen, Chengming Hu, Pengyi Liao, Yiran Pang

llms

One email a morning, machine-written

Related stories

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics

Do Methods Support the Claims? Intra-Paper Verification for Peer Review

Contemporary AI lacks the imagination to diverge or negate in science

Measuring the Creativity of Frontier LLMs in Automated Research