arXiv AI By Yueran Tian, Huizhi Liang

NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers

Read the original on arXiv AI →

NoveltyScope is a prototype that evaluates the novelty of a scientific paper by comparing it with prior work and generating a structured report and a five‑level novelty score. It extracts the core task and up to two author‑claimed contributions, retrieves relevant passages from a local corpus, and uses a four‑role multi‑agent system (Prosecutor, Defender, Judge, Critic) to assess novelty, allowing a single revision of the Judge’s score. Evaluation on 50 RINoBench samples shows that multi‑agent scoring with Gemini or Qwen3 achieves lower mean absolute errors (1.26–1.16) than single‑LLM scoring (1.48–1.22) and outperforms a vector‑similarity baseline (1.68).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 12

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

NovGauge is a new benchmark designed to diagnose large language models’ ability to assess scientific paper novelty. It contains 619 paper pairs and 50 multi-paper sets, each labeled along three dimensions—task, problem, and method—by experts from ICLR reviewer overlap claims and survey co-citations. The study evaluates 18 LLMs, revealing high hallucination rates and weak evidence grounding, with the best model achieving only 43‑72% verified F1 across dimensions.

By Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing, Zhonghan Yue, Jiayi Chen, Shiqiang Wu, Shaofan Liu, Yue Zhang, Yuankai Ying, Yang Shi, Tao Gui, Qi Zhang, Xuanjing Huang