arXiv AI
Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 22
Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics
arXiv:2609.23264v1 Announce Type: new Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
arXiv Computation and Language
Aug 24
Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation
Large language models (LLMs) are increasingly used as judges for code evaluation, assessing correctness without reference implementations. This study investigates whether LLM judges can fairly evaluate semantically equivalent code that differs in superficial aspects such as variable names, comments, or formatting. The authors define six types of potential bias, conduct experiments across five programming languages and multiple LLMs, and find that all tested judges exhibit both positive and negative biases, leading to inflated or unfairly low scores even when prompted to generate test cases.
arXiv AI
Jun 26
ReportLogic: Evaluating Logical Quality in Deep Research Reports
arXiv:2602. 18446v2 Announce Type: replace-cross Abstract: Users increasingly rely on Large Language Models (LLMs) for Deep Research, using them to synthesize diverse sources into structured reports that support understanding and action.
Hugging Face Trending Papers
Aug 6
Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.
arXiv AI
Aug 11
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
arXiv:2608. 08975v1 Announce Type: cross Abstract: As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions.
arXiv AI
Aug 28
From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
The paper introduces MCR-Bench, a benchmark for realistic multi‑round code review that includes 2,269 real‑world tasks across five programming languages, each annotated with fine‑grained defect information and dynamic state labels. Experiments with mainstream large language models show limited overall performance, especially as interaction rounds increase, and reveal that model accuracy varies by defect type and severity. Error analysis identifies key failure mechanisms such as cross‑round temporal misalignment and insufficient long‑range memory.