The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
By Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Bego\~na Altuna
arXiv:2609.00241v1 Announce Type: new
Abstract: Long documents often distribute important information across extensive narrative passages and multiple tables, making faithful summarization particular...
By Meng Zhou, Wenhao You, Wei Yuan
arXiv:2608. 03655v1 Announce Type: cross Abstract: Abstractive summarization models remain vulnerable to factual inconsistency, redundancy, and weak length control.
By Zeyu Wang, Guanghua Wang, Meng Xu
arXiv:2606. 17350v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have enabled the generation of high-quality prose, yet the question of whether these models are capable of generating diverse outputs remains contested.
By Thennal DK, Hans Ole Hatzel
arXiv:2607. 21010v1 Announce Type: new Abstract: Zero-shot summarization using Large Language Models (LLMs) has significantly advanced the abstractive summarization task by producing coherent and fluent summaries.
By Vasudha Bhatnagar, Purnima Bindal, Vikas Kumar, Raj Kumari Bahl
The paper investigates how large language models (LLMs) engage with long-form narratives by comparing their generated novel summaries to human-authored ones. Researchers align sentences from 150 human-written summaries to specific chapters, highlighting the challenge of this alignment task and the complexity of summarization. They find stylistic differences and that LLMs tend to focus more on the ends of texts, suggesting insights into why models may struggle with narrative comprehension.
By Rebecca M. M. Hicke, Sil Hamilton, David Mimno, Ross Deans Kristensen-McLachlan
The paper introduces Gavel, a framework for evaluating large language models (LLMs) on long-context legal summarization tasks. Gavel includes a reference-based component (Gavel-Ref) with checklist, residual-fact, and writing-style checks, and a reference-free component (Gavel-Agent) that assesses factual coverage directly from source documents. Experiments on 12 frontier LLMs reveal that models tend to omit key information more than hallucinate, perform well on simple checklist items but struggle with rare, complex items, and their performance degrades with longer cases. Gavel-Agent cuts token usage by at least 36% compared to traditional methods while maintaining competitive accuracy, and it also generalizes effectively to the medical domain.
By Yao Dou, Benjamin Mamut, Wei Xu
Scientific long-document summarization datasets commonly treat author-written abstracts as gold reference summaries, although their quality and alignment with the source article vary. At the same time, publicly available scientific summarization datasets remain limited in scale and structure for modern long-context models.
arXiv:2607. 10806v1 Announce Type: cross Abstract: Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE.
By Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala
arXiv:2601.03418v3 Announce Type: replace
Abstract: Trustworthy clinical summarization requires every claim to be traceable to its evidence, yet existing attribution often resolves only to the senten...
By Bohao Chu, Hendrik Damm, Tabea M. G. Pakull, Sameh Frihat, Georg Lodde, Elisabeth Livingstone, Christoph M. Friedrich, Norbert Fuhr
The paper introduces CARPAS, a new task that dynamically refines user-provided aspects for aspect-based summarization in large language models (LLMs). It presents three new datasets and evaluates four prompting strategies, finding that LLMs tend to over-generate aspects, leading to overly long and misaligned summaries. To address this, the authors propose a two-stage framework that first generates lightweight scope guidance before aspect refinement and summarization, which improves focus, reduces over-generation, and enhances performance across all datasets.
By Yong-En Tian, Yu-Chien Tang, An-Zi Yen, Wen-Chih Peng
arXiv:2609.22603v1 Announce Type: new
Abstract: Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metr...
By Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger, Sepideh Kharaghani, Ying Zhang