The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
By Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Bego\~na Altuna
arXiv:2607. 18983v1 Announce Type: cross Abstract: We present AutoJourn, a demonstration system for multi-perspective news generation and bias-aware evaluation using large language models (LLMs).
By Himel Ghosh, Ahmed Mosharafa, Georg Groh
arXiv:2606. 08000v1 Announce Type: cross Abstract: The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem.
By Dongqi Liu, Chenxi Whitehouse, Zheng Zhao, Zhuchen Cao, Jian Li, Yabiao Wang
arXiv:2611.06812v1 Announce Type: new
Abstract: With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating co...
By Man-Ling Sung, Hemanth Kandula, Jeff Ma, William Hartmann, Matthew Snover
The paper introduces PromptNCE, a zero‑shot method that uses large language models to estimate pointwise mutual information (PMI) by framing conditional probability estimation as a contrastive task with an explicit OTHER category. The authors benchmark PromptNCE against four other prompting‑based estimators on three human‑annotated datasets, finding that PromptNCE achieves the best conditional probability estimates and Spearman correlations up to 0.78 for full PMI. A case study demonstrates the method’s utility for scoring student knowledge summaries in low‑data settings, and the authors release code and prompts for reproducibility.
By Juliette Woodrow, Chris Piech
We present AutoJourn, a demonstration system for multi-perspective news generation and bias-aware evaluation using large language models (LLMs). The system tackles three core challenges in responsible automated journalism: extracting diverse perspectives from unstructured social media discussions, generating summaries that preserve viewpoint diversity, and detecting or mitigating bias in AI-generated news.
arXiv:2608. 03655v1 Announce Type: cross Abstract: Abstractive summarization models remain vulnerable to factual inconsistency, redundancy, and weak length control.
By Zeyu Wang, Guanghua Wang, Meng Xu
arXiv:2609.22603v1 Announce Type: new
Abstract: Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metr...
By Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger, Sepideh Kharaghani, Ying Zhang
The paper introduces the Conformal Relevance framework, which employs in-context learning example curation and ensembling to generate a score function that preserves coverage while enhancing conciseness for NLP tasks such as summarization and extractive question answering. Unlike previous methods that rely on labor-intensive, task‑specific prompt engineering, this approach requires minimal manual input. The authors validate the framework across seven NLP tasks and provide a theoretical analysis of how diversity in ensembled conformal scores can improve worst‑case sentence scores, including a saturation bound on ensemble gains.
By Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara, Kin Kwan Leung, Jesse C. Cresswell
arXiv:2512. 09066v2 Announce Type: replace-cross Abstract: Reliable assessment of the abilities of large audio language models (LALMs) is essential to advancing the state of the art.
By \v{S}imon Sedl\'a\v{c}ek, Sara Barahona, Bolaji Yusuf, Laura Herrera-Alarc\'on, Santosh Kesiraju, Cecilia Bola\~nos, Alicia Lozano-Diez, Sathvik Udupa, Fernando L\'opez, Allison Ferner, Ramani Duraiswami, Jan \v{C}ernock\'y
arXiv:2610.09660v1 Announce Type: new
Abstract: Automatic Short Answer Scoring (ASAS) requires models that can score student responses against question-specific criteria while remaining efficient and...
By Zhifan Sun, Sebastian Gombert, Fabian Zehner, Leon Camus, Longwei Cong, Hendrik Drachsler
The paper introduces CARPAS, a new task that dynamically refines user-provided aspects for aspect-based summarization in large language models (LLMs). It presents three new datasets and evaluates four prompting strategies, finding that LLMs tend to over-generate aspects, leading to overly long and misaligned summaries. To address this, the authors propose a two-stage framework that first generates lightweight scope guidance before aspect refinement and summarization, which improves focus, reduces over-generation, and enhances performance across all datasets.
By Yong-En Tian, Yu-Chien Tang, An-Zi Yen, Wen-Chih Peng