NovGauge is a new benchmark designed to diagnose large language models’ ability to assess scientific paper novelty. It contains 619 paper pairs and 50 multi-paper sets, each labeled along three dimensions—task, problem, and method—by experts from ICLR reviewer overlap claims and survey co-citations. The study evaluates 18 LLMs, revealing high hallucination rates and weak evidence grounding, with the best model achieving only 43‑72% verified F1 across dimensions.
By Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv:2606. 12071v1 Announce Type: cross Abstract: LLMs are increasingly used to generate and judge scientific ideas.
By Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria
arXiv:2603.20884v4 Announce Type: replace
Abstract: To alleviate the heavy burden of paper screening, researchers increasingly rely on existing AI agents, such as AI reviewers or DeepResearch, for pa...
By Jiajun Hou, Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Xiaopeng Ke, Derek F. Wong, Min Zhang
arXiv:2604. 15145v2 Announce Type: replace Abstract: The rigorous evaluation of the novelty of a scientific paper is, even for human scientists, a challenging task.
By Miri Liu, ChengXiang Zhai
arXiv:2607. 26066v1 Announce Type: cross Abstract: The growing volume of scientific submissions has motivated interest in using large language models (LLMs) to assist peer review.
By Ranjitha Shivaprasad Ballakuraya, Arash Mahyari, Ashok Srinivasan
arXiv:2610.02022v2 Announce Type: replace
Abstract: AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models....
By Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope
The paper proposes a new method for measuring the novelty of biomedical research by calculating latent distances between knowledge units—specifically MeSH terms—using three types of relationships: network, semantic, and hierarchical. It demonstrates that each relationship captures distinct distances and that combining all three yields a more accurate novelty assessment than existing metrics. Validation on a large PLoS ONE dataset and a H1 Connect dataset shows stronger alignment with peer judgments compared to prior indicators.
By Yi Zhao, Heng Zhang, Yuzhuo Wang, Wenqing Wu, Tong Bao, Chengzhi Zhang
arXiv:2610.33644v1 Announce Type: new
Abstract: Research ideas are always proposed under resource constraints: a plan must fit the available compute, API credit, and time. Yet evaluations of research...
By Shunchao Zhou, Chang Yang, Jingrui Wang, Shuxin Zhuang, Rundong Wang, Runsheng Yu, Xinrun Wang
The paper introduces Think‑Probe‑Respond (TPR), a lightweight method to improve large language models’ ability to judge the novelty of research ideas. It identifies a systematic bias where models tend to label ideas as "medium novel" despite generating human‑like rationales, and shows that probing hidden states during reasoning and conditioning the final response on these probes boosts novelty judgment accuracy by 22.30%. TPR effectively reduces the medium‑novelty bias across strong baseline models.
By Tim Schopf, Tobias Schreieder, Akiko Aizawa
PatentAgentBench is a new benchmark that tests large language model agents on 510 real-world patent tasks across eight suites, including novelty search, freedom‑to‑operate, design FTO, invalidity search, claim charting, patent drafting, office‑action response, and patent translation. The benchmark uses curated patent records and expert‑reviewed annotations to evaluate task‑specific metrics, requiring agents to search patent collections without pre‑supplied candidate pools and to produce high‑quality drafting and response work. Seven LLMs were evaluated on four agent harnesses, revealing that current systems still struggle with novelty identification, risk assessment, and understanding practitioner intent throughout patent workflows.
By Yijie Hu, Wei Li, Junjie Zhou, Zhiqi Jiang, Yi Jin, Licong Xu, Xinkai Sun, Kai Duan, Lisha Zhang
The paper introduces ClaimNov, a claim‑level novelty detection module designed to improve AI scientific discovery pipelines. By evaluating novelty at the proposal and claim levels using four datasets of 1,060 proposals, the authors show that existing checks either reject too many novel ideas or fail to isolate non‑novel parts of a proposal. ClaimNov achieves a 97% detection rate of disguised papers with only a 1.5% false‑alarm rate, significantly reducing false positives compared to prior methods and preserving proposals that build on established work.
By Miri Liu, Xiayu Zhao, Encheng Xie, ChengXiang Zhai
PatEx-Gym is a new framework for developing and evaluating agents that perform patent examination. It includes PatEx-Bench, a benchmark of 1,000 realistic tasks with large retrieval corpora, and PASS, an annotation‑free method for synthesizing search trajectories. A 4B‑parameter PatEx‑Agent trained on these synthetic trajectories provides a lightweight baseline that competes with much larger general‑purpose models, yet prior‑art discovery remains challenging.
By Qiyao Wang, Xinyi Chen, Hongbo Wang, Jiayuan Lu, Jianhang Li, Minghuan Tan, Yuan Lin, Min Yang