arXiv:2607. 27209v1 Announce Type: cross Abstract: Peer review at ML conferences increasingly relies on reviewer scores as the primary decision instrument.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
The paper investigates whether peer review quality is declining by developing a framework for evidence-based comparison across venues and time. It applies this framework to major AI and machine learning conferences—ICLR, NeurIPS, and ACL—documenting diverse review formats and proposing a multi-dimensional schema that combines LLM-based and lightweight metrics. The study finds no consistent decline in median review quality over time, offering alternative explanations and recommendations for future empirical research.
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
The paper introduces an observational estimator to quantify how much peer‑review decisions would change if a paper were reviewed by a different set of reviewers, using public review data from ICLR 2017‑2025. The estimator decomposes scores into paper quality and reviewer noise via a Bayesian ordered‑probit model, maps scores to decisions with a logistic model, and simulates independent committees to estimate disagreement rates, finding 23‑30% disagreement at committee size two and 18‑24% at size four. Calibration against NeurIPS 2021 data shows close agreement, and longitudinal analysis reveals no robust trend in reviewer noise over the period, though specific years (2020, 2021) exhibit higher flip rates due to scale compression and low signal‑to‑noise. whyItMatters":"The study provides a scalable, data‑driven method to assess the reliability of conference accept/reject decisions, offering insights into reviewer noise and potential improvements in review processes."
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.
arXiv:2607. 15388v1 Announce Type: new Abstract: Many math- and science-oriented agent systems use hierarchical designs with specialized reviewer roles, assuming that a dedicated review stage should help turn wrong candidates into correct ones.