Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment

The study evaluates five large language model (LLM) review paradigms across different input formats and models on 232 ICLR 2025 submissions. Direct absolute rating aligns with human rankings but tends to over‑accept papers, while recalibration improves agreement and reduces API calls. Abstract‑only inputs outperform full‑paper summarization, and model choice is as influential as the paradigm itself, yet none of the LLM methods surpasses a simple always‑reject baseline in accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• nlp