arXiv:2606. 19749v1 Announce Type: new Abstract: A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated.
By Dang Nguyen, Wanqing Hao, Yanai Elazar, Chenhao Tan
FLAWS is a new benchmark that tests large language models’ ability to find and pinpoint errors that invalidate key claims in scientific papers. The dataset contains 713 paper‑error pairs created by inserting claim‑invalidating mistakes into peer‑reviewed articles using LLMs, and it includes an automated metric for measuring error detection and localization. Five state‑of‑the‑art LLMs were evaluated, with GPT‑5 achieving the highest accuracy of 39.1% when selecting the top 10 candidate error texts.
By Sarina Xi, Vishisht Rao, Justin Payan, Nihar B. Shah
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
By Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman
arXiv:2506. 08134v4 Announce Type: replace Abstract: Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale.
By Qiyao Wei, Samuel Holt, Jing Yang, Markus Wulfmeier, Mihaela van der Schaar
arXiv:2603. 20450v2 Announce Type: replace-cross Abstract: A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews.
By Rounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan, Nihar B Shah, Danish Pruthi
The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.
By Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat