The paper introduces an author-facing large language model (LLM) system that generates a broad set of atomic concerns about a manuscript and compresses them into a concise report, aiming to provide early peer‑review feedback. Evaluations on 3,398 ICLR 2026 submissions show that the system covers 44.9% of historical reviewer issues on a diagnostic set, rising to 78.7% strict coverage and 84.9% seriousness‑weighted coverage after deduplication and refill, using 3.6× more requests and 5.2× more tokens. Ablation studies reveal that representative selection and matcher sensitivity are key factors limiting the compression quality.
By Pouya Parsa, Amin Rezaei
The paper investigates whether peer review quality is declining by developing a framework for evidence-based comparison across venues and time. It applies this framework to major AI and machine learning conferences—ICLR, NeurIPS, and ACL—documenting diverse review formats and proposing a multi-dimensional schema that combines LLM-based and lightweight metrics. The study finds no consistent decline in median review quality over time, offering alternative explanations and recommendations for future empirical research.
By Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych
The paper introduces an observational estimator to quantify how much peer‑review decisions would change if a paper were reviewed by a different set of reviewers, using public review data from ICLR 2017‑2025. The estimator decomposes scores into paper quality and reviewer noise via a Bayesian ordered‑probit model, maps scores to decisions with a logistic model, and simulates independent committees to estimate disagreement rates, finding 23‑30% disagreement at committee size two and 18‑24% at size four. Calibration against NeurIPS 2021 data shows close agreement, and longitudinal analysis reveals no robust trend in reviewer noise over the period, though specific years (2020, 2021) exhibit higher flip rates due to scale compression and low signal‑to‑noise.
whyItMatters":"The study provides a scalable, data‑driven method to assess the reliability of conference accept/reject decisions, offering insights into reviewer noise and potential improvements in review processes."
By Feilian Huang
arXiv:2609.25046v1 Announce Type: new
Abstract: Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely...
By Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour, Soroush Sadeghian, Radin Cheraghi, Negar Arabzadeh, Ebrahim Bagheri
arXiv:2609.05947v1 Announce Type: new
Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of g...
By Siming Yuan, Xueyi Zhang, Wangze Ni, Tianfang Xiao, Shimin Di, Jia Zhu, Zhuoren Jiang, Rong Tan, Lei Chen, Kui Ren
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
By Costa Georgantas
arXiv:2606.13349v2 Announce Type: replace
Abstract: Effective peer review requires adaptive investigation that responds to evidence emerging from each paper. Existing LLM-based reviewers largely orga...
By Haishuo Fang, Yue Feng, Iryna Gurevych
arXiv:2606. 19749v1 Announce Type: new Abstract: A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated.
By Dang Nguyen, Wanqing Hao, Yanai Elazar, Chenhao Tan
PremiseCheck is a framework that improves fact‑checking of peer‑review comments by first having an LLM rewrite each comment into explicit premises without reading the paper, and then using a decision model to verify each premise against the paper. This approach achieves accuracy comparable to GPT‑5.5 across NLP, machine learning, and natural science reviews while cutting costs roughly in half and reducing latency, especially in NLP reviews where it cuts cost to a quarter and latency to less than half. The system is built on the ReviewFact benchmark, which labels factual errors in review comments and measures cost, latency, and accuracy.
By Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun
FLAWS is a new benchmark that tests large language models’ ability to find and pinpoint errors that invalidate key claims in scientific papers. The dataset contains 713 paper‑error pairs created by inserting claim‑invalidating mistakes into peer‑reviewed articles using LLMs, and it includes an automated metric for measuring error detection and localization. Five state‑of‑the‑art LLMs were evaluated, with GPT‑5 achieving the highest accuracy of 39.1% when selecting the top 10 candidate error texts.
By Sarina Xi, Vishisht Rao, Justin Payan, Nihar B. Shah
The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.
By Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat
AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities.