The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper examines the intermediate‑entropy regime of natural language generation, where a single input can have multiple valid outputs such as in peer‑review generation and image captioning. It introduces a target‑side consolidation method that merges all references into a single target preserving every distinct information unit, and presents the MERC‑67K dataset of 36,537 papers and 31,014 images with both original and consolidated references. Human evaluations show that models trained on consolidated references achieve higher coverage and overall quality scores, outperforming multi‑reference baselines across several large language models while using fewer individual references during training.
Large language models are increasingly used as automated reviewers in scientific evaluation, creating a recursive feedback loop where later reviewers learn from earlier model-generated judgments. A study using Llama 3.1 8B fine‑tuned on ICLR reviews shows that incorporating synthetic reviews compresses rating distributions and reduces semantic diversity, a phenomenon termed scientific‑judgment collapse. To counter this, the authors introduce TrustReviewer, an open‑source LLM system that curates training data and applies paired activation steering at test time to preserve judgment diversity and improve recommendation alignment.
The paper introduces a verification‑centric framework and benchmark for evaluating large language models (LLMs) as assistants in peer review, focusing on error detection rather than imitation of human reviews. It presents a scalable benchmark that injects synthetic errors into conference papers to test LLMs’ ability to spot logical contradictions, and proposes a Multi‑Layered Review (MLR) framework that emphasizes deep manuscript comprehension before generating reviews. Experiments show the approach aligns well with human review scores, excels at error detection, and offers complementary reviewer insights, while also revealing vulnerabilities to adversarial manipulation.
OARelatedWork is a large‑scale dataset for generating entire related‑work sections from open‑access papers, comprising 94,450 papers across multiple domains. It includes full texts of cited works and is designed so every source paper is available in full, enabling controlled evaluation of full‑text related‑work generation. The paper benchmarks various models, shows that large language models struggle with massive full‑text contexts, and introduces a statement‑level evaluation framework to better assess long‑form structured outputs.
arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.
arXiv:2608. 03859v1 Announce Type: cross Abstract: Large language models (LLMs) pose challenges to academic integrity and peer review.