The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
arXiv:2607. 21010v1 Announce Type: new Abstract: Zero-shot summarization using Large Language Models (LLMs) has significantly advanced the abstractive summarization task by producing coherent and fluent summaries.
arXiv:2606. 08000v1 Announce Type: cross Abstract: The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem.
arXiv:2608. 04307v1 Announce Type: cross Abstract: Text summarization is deceptively difficult.
The paper introduces CARPAS, a new task that dynamically refines user-provided aspects for aspect-based summarization in large language models (LLMs). It presents three new datasets and evaluates four prompting strategies, finding that LLMs tend to over-generate aspects, leading to overly long and misaligned summaries. To address this, the authors propose a two-stage framework that first generates lightweight scope guidance before aspect refinement and summarization, which improves focus, reduces over-generation, and enhances performance across all datasets.
arXiv:2609.00241v1 Announce Type: new Abstract: Long documents often distribute important information across extensive narrative passages and multiple tables, making faithful summarization particular...