The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The study evaluates five large language model (LLM) review paradigms across different input formats and models on 232 ICLR 2025 submissions. Direct absolute rating aligns with human rankings but tends to over‑accept papers, while recalibration improves agreement and reduces API calls. Abstract‑only inputs outperform full‑paper summarization, and model choice is as influential as the paradigm itself, yet none of the LLM methods surpasses a simple always‑reject baseline in accuracy.
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
The paper compares Jev with nine language models on the ContractNLI task, assessing inference cost, response time, average correctness, and correctness under repeated requests. Controlled experiments vary hypothesis visibility, requested outputs, and output order while keeping the contract and target judgment fixed. Jev achieves the lowest cost and median response time, whereas hosted language models show higher baseline accuracy, but rankings differ when evaluating correctness across all conditions and repeats.
FLAWS is a new benchmark that tests large language models’ ability to find and pinpoint errors that invalidate key claims in scientific papers. The dataset contains 713 paper‑error pairs created by inserting claim‑invalidating mistakes into peer‑reviewed articles using LLMs, and it includes an automated metric for measuring error detection and localization. Five state‑of‑the‑art LLMs were evaluated, with GPT‑5 achieving the highest accuracy of 39.1% when selecting the top 10 candidate error texts.
The study examines whether AI reviewers’ scores reflect authors’ revisions by analyzing 6,959 paper trajectories from five AI conferences. Major revisions that add empirical evidence and improve framing are identified, yet most reviewers do not change scores for these improvements and agreement on score direction is weak. Reviewer-specific associations exist, but overall predictive power is limited, and brief negative comments can negate any scoring gains from substantive revisions.
The study examines how the rapid turnover of large language models (LLMs) undermines the effectiveness of AI‑detectors used to screen scientific manuscripts. By comparing 4,000 PNAS abstracts rewritten by 23 different LLM versions released between June 2023 and August 2026, the authors show that detectors trained on past versions can fail dramatically at generation boundaries, missing up to 96% of rewrites just after a sharp transition. Even detectors updated to newer versions can overlook rewrites of older models, and a commercial detector missed most rewrites of the newest version while flagging few human‑written abstracts.