arXiv Computation and Language
SAUCE: Summary Analysis Using Conversation Entailment
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Aug 28
Evaluating Language Models in Realistic Conversational Contexts
The paper introduces UPHELD, a large benchmark of human-to-human dialogues written by professional script writers, featuring realistic turn densities and over 36,000 per-turn human annotations. It evaluates existing automatic metrics and LLM-as-a-judge methods, finding them unreliable against expert human judgment. Using UPHELD, the authors develop a Mixture-of-Judges framework that improves correlation with human assessments by about 30%.
arXiv AI
Jun 9
Summarization is Not Dead Yet
arXiv:2606. 08000v1 Announce Type: cross Abstract: The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem.
arXiv AI
Sep 3
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.
Hugging Face Trending Papers
Sep 8
EviSI: An Evaluation Agent for Simultaneous Interpreting
EviSI is a large language model evaluation agent designed for simultaneous speech-to-speech translation. It adapts Multidimensional Quality Metrics to assess semantic fidelity and oral expression, using shared source evidence and deterministic scoring. In English‑to‑Chinese, EviSI achieves a mean Kendall agreement of 0.707 with human system rankings, outperforming baseline metrics, and shows positive concordance with COMET across five translation directions.
arXiv Computation and Language
Sep 16
EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting
EviSI is an evidence‑based evaluation agent for low‑latency simultaneous speech‑to‑speech translation. It combines Multidimensional Quality Metrics with interpreter‑developed criteria, using shared source evidence to assess four dimensions—Anchor, Event, Logic, and Fluency—while deduplicating verified errors before scoring. On English‑to‑Chinese and Chinese‑to‑English data, EviSI’s rankings correlate strongly with human judgments, outperforming BLEU and COMET, and its multilingual extension maintains these correlations across five language directions.
arXiv Computation and Language
Oct 15
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
The study introduces a zero‑shot benchmark for eight large language models on English and Bengali abstractive summarization, using both lexical/semantic metrics and reference‑free factuality checks. Results show significant cross‑lingual differences: GPT‑5.6 Luna excels in English but not in Bengali, where LLaMA 3.3 70B tops ROUGE and BLEU, while Gemini 2.5 Flash achieves the best Bengali factuality score. The evaluation also reveals that different metrics rank models differently, underscoring the need for multi‑dimensional assessment in low‑resource languages.