The study evaluates five large language model (LLM) review paradigms across different input formats and models on 232 ICLR 2025 submissions. Direct absolute rating aligns with human rankings but tends to over‑accept papers, while recalibration improves agreement and reduces API calls. Abstract‑only inputs outperform full‑paper summarization, and model choice is as influential as the paradigm itself, yet none of the LLM methods surpasses a simple always‑reject baseline in accuracy.
By Fan Huang, Hongyu Zhou
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
By Costa Georgantas
The paper compares Jev with nine language models on the ContractNLI task, assessing inference cost, response time, average correctness, and correctness under repeated requests. Controlled experiments vary hypothesis visibility, requested outputs, and output order while keeping the contract and target judgment fixed. Jev achieves the lowest cost and median response time, whereas hosted language models show higher baseline accuracy, but rankings differ when evaluating correctness across all conditions and repeats.
By Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng, Lei Fan, Xinhua Ji, Cunyuan Zheng, Huangyong Shan, Philip S. Yu, Xue Liu, Yu Chen, Preslav Nakov, Songwei He
FLAWS is a new benchmark that tests large language models’ ability to find and pinpoint errors that invalidate key claims in scientific papers. The dataset contains 713 paper‑error pairs created by inserting claim‑invalidating mistakes into peer‑reviewed articles using LLMs, and it includes an automated metric for measuring error detection and localization. Five state‑of‑the‑art LLMs were evaluated, with GPT‑5 achieving the highest accuracy of 39.1% when selecting the top 10 candidate error texts.
By Sarina Xi, Vishisht Rao, Justin Payan, Nihar B. Shah
The study examines whether AI reviewers’ scores reflect authors’ revisions by analyzing 6,959 paper trajectories from five AI conferences. Major revisions that add empirical evidence and improve framing are identified, yet most reviewers do not change scores for these improvements and agreement on score direction is weak. Reviewer-specific associations exist, but overall predictive power is limited, and brief negative comments can negate any scoring gains from substantive revisions.
By Yaochen Wang, Chenguang Wang, Ming Li, Haomin Zhuang, Xinyue Zeng, Han Chen, Kwesi Adu Cobbina, Dang Nguyen, Dianqi Li, Dawei Zhou, Tianyi Zhou
The study examines how the rapid turnover of large language models (LLMs) undermines the effectiveness of AI‑detectors used to screen scientific manuscripts. By comparing 4,000 PNAS abstracts rewritten by 23 different LLM versions released between June 2023 and August 2026, the authors show that detectors trained on past versions can fail dramatically at generation boundaries, missing up to 96% of rewrites just after a sharp transition. Even detectors updated to newer versions can overlook rewrites of older models, and a commercial detector missed most rewrites of the newest version while flagging few human‑written abstracts.
By Kazuki Nakajima, Takayuki Mizuno
arXiv:2608. 08975v1 Announce Type: cross Abstract: As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions.
By Ming Li, Chenguang Wang, Xirui Li, Xinyue Zeng, Dianqi Li, Peng Shi, Dawei Zhou, Tianyi Zhou
arXiv:2610.31791v1 Announce Type: cross
Abstract: Automated research is largely evaluated by benchmark performance, but the value of research rarely reduces to a single score. Humans judge a work's h...
By Sachin Konan, Jonathan Liu, Zhuang Liu
The paper investigates when it is better to return an existing draft answer or revise it using retrieved evidence in retrieval‑augmented QA systems. By grading both the draft and its candidate revision with the same correctness judge, the authors define a paired effect called recoverability and train policies to predict it before revision. Experiments on 25,870 open‑domain questions show that a recoverability‑based scorer outperforms a draft‑correctness scorer across multiple Llama setups, improving accuracy–revision trade‑offs and closing a significant portion of the oracle gap, though it still applies harmful revisions in a substantial fraction of cases.
By Nicholas Kashani Motlagh, Tim Anderson, Jeremy Gwinnup, Grant Erdmann
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
By Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman
arXiv:2611.06384v1 Announce Type: new
Abstract: Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentat...
By Jatin Agrawal, Radhika Mamidi
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.