arXiv Computation and Language
When Evidence Conflicts: Reliability-aware Meta-review Generation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Oct 9
Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.
arXiv AI
Jul 1
Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance
arXiv:2601. 14171v2 Announce Type: replace Abstract: Writing effective rebuttals is a high-stakes task that demands more than linguistic fluency, as it requires precise alignment between reviewer intent and manuscript details.
arXiv Computation and Language
Sep 22
Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics
arXiv:2609.23264v1 Announce Type: new Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
arXiv Machine Learning
Sep 21
When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
Large language models are increasingly used as automated reviewers in scientific evaluation, creating a recursive feedback loop where later reviewers learn from earlier model-generated judgments. A study using Llama 3.1 8B fine‑tuned on ICLR reviews shows that incorporating synthetic reviews compresses rating distributions and reduces semantic diversity, a phenomenon termed scientific‑judgment collapse. To counter this, the authors introduce TrustReviewer, an open‑source LLM system that curates training data and applies paired activation steering at test time to preserve judgment diversity and improve recommendation alignment.
arXiv Computation and Language
Sep 22
To Consolidate or not to Consolidate? Evaluating the Impact of Consolidation in Multi-Reference Training using Peer Reviews
arXiv:2609.22805v1 Announce Type: new Abstract: Natural language generation (NLG) tasks span the spectrum of conditional entropy, ranging from highly constrained machine translation to open-ended dia...
arXiv Machine Learning
Aug 24
Metag: A dataset to build agentic meta-reviewing capabilities
arXiv:2608.20488v1 Announce Type: new Abstract: AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same...