The study evaluates five large language model (LLM) review paradigms across different input formats and models on 232 ICLR 2025 submissions. Direct absolute rating aligns with human rankings but tends to over‑accept papers, while recalibration improves agreement and reduces API calls. Abstract‑only inputs outperform full‑paper summarization, and model choice is as influential as the paradigm itself, yet none of the LLM methods surpasses a simple always‑reject baseline in accuracy.
By Fan Huang, Hongyu Zhou
arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.
By Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez
arXiv:2608.28626v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly used to generate peer reviews, prompting examination of their capacity for critical evaluation. This st...
By Emad Alharbi
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
By Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman
The paper investigates whether peer review quality is declining by developing a framework for evidence-based comparison across venues and time. It applies this framework to major AI and machine learning conferences—ICLR, NeurIPS, and ACL—documenting diverse review formats and proposing a multi-dimensional schema that combines LLM-based and lightweight metrics. The study finds no consistent decline in median review quality over time, offering alternative explanations and recommendations for future empirical research.
By Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych
Researchers investigated how different large language models (LLMs) convert corporate text into empirical variables, focusing on thirteen measures such as sentiment, management clarity, uncertainty, answer specificity, and climate and political risk. Using seven LLMs to score earnings call transcripts of S&P 500 companies, they found low cross-model rank correlations (average 0.52) and that transcript-level differences across providers explained only 34% of total score variation. The study shows that model choice significantly alters downstream inference, with varying coefficient magnitudes, signs, and statistical significance, and that averaging across providers stabilizes rankings but not score levels.
By Hamid Boustanifar, Sasan Mansouri
arXiv:2610.35254v2 Announce Type: replace
Abstract: The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review worklo...
By Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz
arXiv:2610.35254v1 Announce Type: new
Abstract: The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload o...
By Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann, Ivaxi Sheth, Ruta Binkyte, Koel Dutta Chowdhury, Raouf Kerkouche, Hui-Po Wang, Mario Fritz
arXiv:2608. 07762v1 Announce Type: new Abstract: LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable.
By Sahil Pardasani, Madhusudan Singh
AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities.
arXiv:2608. 12342v1 Announce Type: cross Abstract: Ensuring the accuracy of financial documents is critical for economic analysis, regulatory compliance, and corporate decision-making.
By Ying He, Zhouhong Gu, Zhecheng Hu, Yubo Zhou, Hao Shen, Jiaqing Liang, Zhaoqian Dai, Shuguang Ma, Fei Yu, Yanghua Xiao, Zhixu Li
Researchers find that newer GPT models increasingly rewrite academic paragraphs in a way that weakens or retracts the original authors’ claims—a phenomenon they term "defensive writing." The study shows that this effect grows with GPT version, with GPT‑6‑astra often removing claims outright and adding ungrounded qualifiers when rewriting from evidence sheets. The authors suggest that the models are likely tailoring text for an anticipated reviewer rather than simply correcting overclaiming, and that this defensive style can be amplified when AI writing is combined with AI review.