The paper investigates whether peer review quality is declining by developing a framework for evidence-based comparison across venues and time. It applies this framework to major AI and machine learning conferences—ICLR, NeurIPS, and ACL—documenting diverse review formats and proposing a multi-dimensional schema that combines LLM-based and lightweight metrics. The study finds no consistent decline in median review quality over time, offering alternative explanations and recommendations for future empirical research.
By Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
By Costa Georgantas
The paper introduces an observational estimator to quantify how much peer‑review decisions would change if a paper were reviewed by a different set of reviewers, using public review data from ICLR 2017‑2025. The estimator decomposes scores into paper quality and reviewer noise via a Bayesian ordered‑probit model, maps scores to decisions with a logistic model, and simulates independent committees to estimate disagreement rates, finding 23‑30% disagreement at committee size two and 18‑24% at size four. Calibration against NeurIPS 2021 data shows close agreement, and longitudinal analysis reveals no robust trend in reviewer noise over the period, though specific years (2020, 2021) exhibit higher flip rates due to scale compression and low signal‑to‑noise.
whyItMatters":"The study provides a scalable, data‑driven method to assess the reliability of conference accept/reject decisions, offering insights into reviewer noise and potential improvements in review processes."
By Feilian Huang
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
By Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman
arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.
By Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez
arXiv:2607. 15388v1 Announce Type: new Abstract: Many math- and science-oriented agent systems use hierarchical designs with specialized reviewer roles, assuming that a dedicated review stage should help turn wrong candidates into correct ones.
By Chih-Hsuan Yang, Jingyan Jiang, Vikram Vasudevan, Cheng-Hau Yang, Huihuo Zheng, Le Chen, Eliu A. Huerta, Venkatram Vishwanath, Ian T. Foster, Rajeev Thakur
arXiv:2606. 10159v1 Announce Type: cross Abstract: AI is increasingly used to support scientific peer review, from manuscript screening, reviewer assistance to editorial triage.
By Lin Li, Qi Zhang, Xander Davies, Jianing Qiu, Yarin Gal
AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities.
arXiv:2608.21374v1 Announce Type: new
Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
By Ruotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
arXiv:2608. 03581v1 Announce Type: cross Abstract: AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are.
By Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber, Kry\v{s}tof Ol\'ik, Georg Groh
arXiv:2609.05947v1 Announce Type: new
Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of g...
By Siming Yuan, Xueyi Zhang, Wangze Ni, Tianfang Xiao, Shimin Di, Jia Zhu, Zhuoren Jiang, Rong Tan, Lei Chen, Kui Ren
arXiv:2609.23264v1 Announce Type: new
Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
By Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le, Negar Arabzadeh, Ebrahim Bagheri