arXiv AI
Jul 7
Stop Automating Peer Review Without Rigorous Evaluation
arXiv:2605. 03202v2 Announce Type: replace Abstract: Large language models offer a tempting solution to address the peer review crisis.
Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable
arXiv:2603. 20450v2 Announce Type: replace-cross Abstract: A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews.
Related stories
Hugging Face Trending Papers
Aug 4
AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities.
arXiv AI
Aug 5
AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
arXiv:2608. 03581v1 Announce Type: cross Abstract: AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are.
arXiv AI
3d ago
Policy-Conditioned AI-Use Detection: An Evidentiary Framework for Academic Publishing
The paper introduces a policy‑conditioned AI‑use detection framework for academic publishing, arguing that traditional AI detection tools misalign with venue rules by merely identifying AI‑generated text. Instead, the proposed system treats the governing policy as an explicit input, generating hypotheses, evidence, calibration, and uncertainty rather than binary verdicts. It outlines how to benchmark compliance, evaluate true positive rates at venue‑specified false positive thresholds, and emphasizes the need for structured disclosure, tool routing, and contestable findings.
arXiv AI
Jun 10
Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community
arXiv:2606. 10159v1 Announce Type: cross Abstract: AI is increasingly used to support scientific peer review, from manuscript screening, reviewer assistance to editorial triage.
arXiv AI
Aug 18
Local AI pre-screening for human triple-blind peer review in health sciences
arXiv:2608. 14625v1 Announce Type: cross Abstract: Academic peer review is under mounting strain: NeurIPS 2025 received 21,575 submissions, ICLR 2025 received 11,603, and ICML 2025 received 12,107.
arXiv AI
Aug 19
Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review
The article reports that in July 2025, 18 arXiv manuscripts contained hidden instructions designed to manipulate AI‑assisted peer review, such as covert commands to give only positive reviews. These prompts were concealed using white text and microscopic fonts, and the authors’ reactions ranged from withdrawal to defending the practice as a test of reviewer misuse of large language models. The study identifies four types of hidden prompts, critiques the ineffectiveness of honeypot defenses, and highlights inconsistent publisher policies while calling for controlled AI integration and harmonized guidelines in academic evaluation.
arXiv Computation and Language
Sep 24
How Much Were You Told? Measuring External Information in Peer Reviews
The paper introduces Self‑Conditioning, an unsupervised, information‑theoretic estimator that measures the amount of external information in peer reviews. It compares the likelihood of a review under its original production context with the likelihood when that context is augmented by hints extracted from the review itself. On the IntelLabs benchmark, Self‑Conditioning can perfectly distinguish fully‑delegated reviews from machine‑polished ones, remains largely insensitive to surface rewriting, and shows that increased external input drives scores toward human‑like values, unlike standard ATD baselines.
arXiv AI
Jun 10
Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystem
arXiv:2506. 08134v4 Announce Type: replace Abstract: Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale.
arXiv AI
Sep 4
HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews
HalluPeer is a new benchmark designed to detect hallucinations in scientific peer reviews. It provides aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, and that HalluPeer-defined hallucination patterns occur in real peer reviews.
Hugging Face Trending Papers
Sep 3
HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews
HalluPeer is a new benchmark designed to detect hallucinations in scientific peer reviews. It provides aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Experiments on 12K papers and 38K reviews show that current detectors struggle to distinguish hallucinations from legitimate critique, and real peer reviews contain HalluPeer-defined hallucination patterns, underscoring the need for source-aware verification.
arXiv AI
Jun 17
BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
arXiv:2510. 18003v2 Announce Type: replace-cross Abstract: The convergence of LLM-powered research assistants and AI-based peer review systems creates a critical vulnerability: fully automated publication loops where AI-generated research is evaluated by AI reviewers without human oversight.