The paper introduces an author-facing large language model (LLM) system that generates a broad set of atomic concerns about a manuscript and compresses them into a concise report, aiming to provide early peer‑review feedback. Evaluations on 3,398 ICLR 2026 submissions show that the system covers 44.9% of historical reviewer issues on a diagnostic set, rising to 78.7% strict coverage and 84.9% seriousness‑weighted coverage after deduplication and refill, using 3.6× more requests and 5.2× more tokens. Ablation studies reveal that representative selection and matcher sensitivity are key factors limiting the compression quality.
By Pouya Parsa, Amin Rezaei
The paper investigates whether peer review quality is declining by developing a framework for evidence-based comparison across venues and time. It applies this framework to major AI and machine learning conferences—ICLR, NeurIPS, and ACL—documenting diverse review formats and proposing a multi-dimensional schema that combines LLM-based and lightweight metrics. The study finds no consistent decline in median review quality over time, offering alternative explanations and recommendations for future empirical research.
By Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych
The paper introduces an observational estimator to quantify how much peer‑review decisions would change if a paper were reviewed by a different set of reviewers, using public review data from ICLR 2017‑2025. The estimator decomposes scores into paper quality and reviewer noise via a Bayesian ordered‑probit model, maps scores to decisions with a logistic model, and simulates independent committees to estimate disagreement rates, finding 23‑30% disagreement at committee size two and 18‑24% at size four. Calibration against NeurIPS 2021 data shows close agreement, and longitudinal analysis reveals no robust trend in reviewer noise over the period, though specific years (2020, 2021) exhibit higher flip rates due to scale compression and low signal‑to‑noise.
whyItMatters":"The study provides a scalable, data‑driven method to assess the reliability of conference accept/reject decisions, offering insights into reviewer noise and potential improvements in review processes."
By Feilian Huang
arXiv:2609.25046v1 Announce Type: new
Abstract: Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely...
By Alireza Daghighfarsoodeh, Sajad Ebrahimi, Ali Ghorbanpour, Soroush Sadeghian, Radin Cheraghi, Negar Arabzadeh, Ebrahim Bagheri
arXiv:2609.05947v1 Announce Type: new
Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of g...
By Siming Yuan, Xueyi Zhang, Wangze Ni, Tianfang Xiao, Shimin Di, Jia Zhu, Zhuoren Jiang, Rong Tan, Lei Chen, Kui Ren
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
By Costa Georgantas