arXiv AI

Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric

arXiv AI
Sep 3

Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English

The paper introduces BASSE, a multilingual meta‑evaluation dataset containing 2,040 human‑rated abstractive summaries produced manually or by five LLMs with four prompts. Annotators scored each summary on coherence, consistency, fluency, relevance, and 5W1H using a 5‑point Likert scale. Benchmarking shows proprietary LLM‑judge models best align with human judgments, followed by criteria‑specific automatic metrics, while open‑source judge LLMs perform poorly.

By Jeremy Barnes, Naiara Perez, Alba Bonet-Jover, Bego\~na Altuna
arXiv Machine Learning
Sep 11

E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

E-CONAN introduces Arabic textual entailment and natural inference benchmarks comprising two datasets: E-CONAN-2 (2-way RTE) and E-CONAN-3 (3-way NLI). The datasets are built from automatically-translated pairs, human-validated machine translations, hand-crafted pairs from Arabic teaching books, and rumor-containing news headlines. The authors evaluated nine multilingual pretrained models and five large language models on these benchmarks, demonstrating that E-CONAN offers a more diverse and robust assessment than existing datasets like XNLI and ArNLI.

By Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi
arXiv Machine Learning
Sep 7

Nepali Passport Question Answering: A Low-Resource Dataset for Public Service Applications

The paper introduces a Nepali Question‑Answer dataset focused on passport‑related FAQs to support information retrieval in a low‑resource language. The authors fine‑tune transformer‑based embedding models for semantic similarity and compare them against the BM25 baseline. Their experiments show that fine‑tuned SBERT models outperform BM25, while multilingual E5 embeddings achieve the best overall retrieval performance.

By Funghang Limbu Begha, Praveen Acharya, Bal Krishna Bal
arXiv Machine Learning
Sep 16

Single Document Extractive Summarization using Domination in Hypergraph

The paper proposes a new approach to single-document extractive summarization by constructing a sentence hypergraph where sentences are nodes and keywords or named entities are hyperedges. A greedy algorithm is then used to find a dominating set of this hypergraph, which yields the sentences that compose the summary. The study compares this hypergraph-based method with existing graph-based summarization techniques.

By Aamir Miyajiwala, Aabha Pingle, Sheetal Sonawane, Surajit Kr. Nath