The paper introduces ReviewMap, a framework that distinguishes between two evaluative acts in peer‑review comments: assessment of claims and scholarly standards versus calibration of contribution value and venue fit. Using multi‑model LLM consensus on 500 ICLR 2026 papers, the act layer is reliably automatable (inter‑model κ ≥ 0.75) while the attribute layer still requires human oversight. The study also finds that 17% of reviewer‑paper instances lack panel‑coded calibration, with a higher calibration share in Strengths than Weaknesses, informing guidelines, rebuttal strategies, and discussion dashboards.
Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliab...
The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.
arXiv:2506. 08134v4 Announce Type: replace Abstract: Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale.
arXiv:2605. 03202v2 Announce Type: replace Abstract: Large language models offer a tempting solution to address the peer review crisis.
arXiv:2608.21374v1 Announce Type: new Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
arXiv:2609.23264v1 Announce Type: new Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
arXiv:2609.05947v1 Announce Type: new Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of g...
PaperDoctor is an agent framework that provides evidence‑grounded, actionable feedback for scientific papers before submission. It evaluates writing, layout, references, code, theory, prior work, and experiments through a three‑layer hierarchical system, linking each critique to specific evidence and revision suggestions. The system selectively rebuilds and reruns experiments to uncover reproducibility gaps, and an interactive interface lets authors explore findings tied to their manuscript.
arXiv:2607. 22553v1 Announce Type: cross Abstract: Peer review is an essential process in scientific research, yet the growing workload has made its automation increasingly necessary.
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.
arXiv:2606. 01013v1 Announce Type: new Abstract: Research is advancing faster than ever with artificial intelligence (AI); and so are the corresponding research papers.
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.