arXiv Computation and Language
SAGE: Schema-Guided LLMs for Grant Review
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Aug 18
FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification
arXiv:2604. 04074v4 Announce Type: replace Abstract: Large language model (LLM)-based reviewing systems typically assess manuscripts in isolation, leaving literature- and code-dependent claims difficult to verify.
arXiv AI
Oct 23
Learning Checklists to Reduce Lottery Effects in Scientific Peer Review
The paper introduces a method to mitigate reviewer lottery error in scientific peer review by generating checklists of criticisms from historical review data. These checklists are organized into a tree of coarsenings, and the optimal coarsening is chosen to maximize correlated agreement among reviewers. Experiments on NeurIPS 2021 and ICLR 2022 data show that the learned checklists significantly reduce mean squared error in simulated and human review settings.
arXiv AI
Jun 4
Knowledge Index of Noah's Ark
arXiv:2606. 05104v1 Announce Type: new Abstract: Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consensus; and unaudited ranking instability under bounded test budgets.
arXiv AI
Aug 25
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
arXiv:2608.21374v1 Announce Type: new Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
Hugging Face Trending Papers
Aug 6
Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.
arXiv AI
Aug 24
ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib
ProofJudge is an agentic large language model that evaluates the quality of formal proofs in Lean 4 beyond mere correctness. It scores proofs on five dimensions—library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions—using tool access to the relevant repository commit. The system was tested on 218 Mathlib declarations, achieving alignment with human reviewers between 63.5% and 80.8% and releasing its harness, dataset, and traces for open research.