PremiseCheck is a framework that improves fact‑checking of peer‑review comments by first having an LLM rewrite each comment into explicit premises without reading the paper, and then using a decision model to verify each premise against the paper. This approach achieves accuracy comparable to GPT‑5.5 across NLP, machine learning, and natural science reviews while cutting costs roughly in half and reducing latency, especially in NLP reviews where it cuts cost to a quarter and latency to less than half. The system is built on the ReviewFact benchmark, which labels factual errors in review comments and measures cost, latency, and accuracy.
arXiv:2606. 19749v1 Announce Type: new Abstract: A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated.
FLAWS is a new benchmark that tests large language models’ ability to find and pinpoint errors that invalidate key claims in scientific papers. The dataset contains 713 paper‑error pairs created by inserting claim‑invalidating mistakes into peer‑reviewed articles using LLMs, and it includes an automated metric for measuring error detection and localization. Five state‑of‑the‑art LLMs were evaluated, with GPT‑5 achieving the highest accuracy of 39.1% when selecting the top 10 candidate error texts.
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
arXiv:2506. 08134v4 Announce Type: replace Abstract: Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale.
arXiv:2603. 20450v2 Announce Type: replace-cross Abstract: A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews.
The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.
The paper introduces Self‑Conditioning, an unsupervised, information‑theoretic estimator that measures the amount of external information in peer reviews. It compares the likelihood of a review under its original production context with the likelihood when that context is augmented by hints extracted from the review itself. On the IntelLabs benchmark, Self‑Conditioning can perfectly distinguish fully‑delegated reviews from machine‑polished ones, remains largely insensitive to surface rewriting, and shows that increased external input drives scores toward human‑like values, unlike standard ATD baselines.
As autonomous AI agents take on every stage of scientific inquiry, research output is expanding far beyond human review capacity. Yet scientific communication still relies on natural-language prose: a...
arXiv:2609.37216v1 Announce Type: cross Abstract: Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers...
AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities.
arXiv:2608. 03581v1 Announce Type: cross Abstract: AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are.
arXiv:2606.13349v2 Announce Type: replace Abstract: Effective peer review requires adaptive investigation that responds to evidence emerging from each paper. Existing LLM-based reviewers largely orga...