The study examines whether AI reviewers’ scores reflect authors’ revisions by analyzing 6,959 paper trajectories from five AI conferences. Major revisions that add empirical evidence and improve framing are identified, yet most reviewers do not change scores for these improvements and agreement on score direction is weak. Reviewer-specific associations exist, but overall predictive power is limited, and brief negative comments can negate any scoring gains from substantive revisions.
arXiv:2609.14738v1 Announce Type: new Abstract: Automated reviewing systems are increasingly evaluated based on the quality of the reviews they produce. Yet a review is only useful if acting on it le...
PaperDoctor is an agent framework that provides evidence‑grounded, actionable feedback for scientific papers before submission. It evaluates writing, layout, references, code, theory, prior work, and experiments through a three‑layer hierarchical system, linking each critique to specific evidence and revision suggestions. The system selectively rebuilds and reruns experiments to uncover reproducibility gaps, and an interactive interface lets authors explore findings tied to their manuscript.
arXiv:2608. 08975v1 Announce Type: cross Abstract: As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions.
arXiv:2606. 10159v1 Announce Type: cross Abstract: AI is increasingly used to support scientific peer review, from manuscript screening, reviewer assistance to editorial triage.
As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions.
The paper introduces EquiReview‑R, an AI‑assisted review system that treats omission and over‑critique as distinct risks and refines a structured concern set using evidence‑linked reasoning. It demonstrates that more criticism does not guarantee a better review, showing that many high‑recall reviews lack definitive evidence for concerns and that revision before further search is essential. On a held‑out set of papers, EquiReview‑R meets non‑inferiority for major omission, cuts major over‑critique from 15.5 % to 8.1 %, and stops on 52.4 % of papers, with gains attributed to revision rather than extra inference.
arXiv:2609.05947v1 Announce Type: new Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of g...
arXiv:2607. 29516v1 Announce Type: cross Abstract: AI coding agents are generating code at volumes that exceed the capacity of traditional peer review.
arXiv:2605. 03202v2 Announce Type: replace Abstract: Large language models offer a tempting solution to address the peer review crisis.
Large language models are increasingly used as automated reviewers in scientific evaluation, creating a recursive feedback loop where later reviewers learn from earlier model-generated judgments. A study using Llama 3.1 8B fine‑tuned on ICLR reviews shows that incorporating synthetic reviews compresses rating distributions and reduces semantic diversity, a phenomenon termed scientific‑judgment collapse. To counter this, the authors introduce TrustReviewer, an open‑source LLM system that curates training data and applies paired activation steering at test time to preserve judgment diversity and improve recommendation alignment.
arXiv:2608. 03581v1 Announce Type: cross Abstract: AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are.
arXiv:2608.20488v1 Announce Type: new Abstract: AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same...