arXiv Computation and Language By Dayam Nadeem, Safdar Mustafa, Ameen Ur Rehman, Shahab Saquib Sohail, Mohammad Nadeem, Usman Naseem

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs

Read the original on arXiv Computation and Language →

The paper presents a bilingual (English/Arabic) benchmark for evaluating how well open-weight, locally deployable language models can accurately recall specific Hadith texts. Five models—Mistral‑7B, Gemma3‑4B, Jais‑7B, ALLaM‑7B, and Phi4‑mini‑3.8B—were tested on 3,000 Hadiths per language across six canonical Sunni collections, with each model given only a bibliographic reference and asked to reproduce the exact text. None of the 30,000 responses achieved both high lexical and semantic fidelity, and Arabic outputs performed uniformly worse than English, indicating that these models, without retrieval augmentation, cannot reliably provide verifiable religious scripture.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Aug 24

Hadith computational science in the age of large language models: a critical narrative review

The article reviews how transformer models, retrieval‑grounded pipelines, and large language models are reshaping hadith computational science. It critically evaluates existing literature, highlighting uneven progress: expanded data resources and mature segmentation tasks, yet persistent issues such as narrow corpora, weak benchmark comparability, and limited reproducibility. The authors argue that hadith computation should be viewed as an evidence‑infrastructure problem requiring knowledge integration, provenance, and expert supervision, and they propose a research agenda to strengthen the field’s methodological rigor.

By Md. Ashraful Haque (Greentech Apps Foundation, United Kingdom), Riasat Islam (Greentech Apps Foundation, United Kingdom, Queen Mary University of London, London, United Kingdom)
Hugging Face Trending Papers
Jul 22

HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer.

arXiv Computation and Language
4d ago

Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

The paper introduces BILAL-BENCH, a 240-question benchmark designed to evaluate open-ended Islamic-source fabrication in Bengali. It assesses models on Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial prompts, using a rubric that separates source errors from assertiveness. Seven open-weight models were tested, revealing that most confidently fabricate on 79.3–95.3% of source-production questions, with hadith provenance being the weakest area and no clear improvement with larger model size.

By Sabah Nushra, Md Kishor Morol
arXiv AI
Aug 24

Ansari: A Retrieval-Grounded Islamic AI Assistant -- Architecture, Deployment, and Lessons from 140,000 Conversations

Ansari is a retrieval‑grounded Islamic AI assistant that has handled over 140,000 conversations in more than 25 languages since June 2023. It uses an agentic retrieval loop where a language model searches authenticated Islamic corpora—including the Qur’an, hadith collections, fiqh encyclopedias, and tafsir sources—and answers only based on retrieved content, providing citations for verification. The paper details Ansari’s architecture, multi‑platform deployment, evaluation results (including top performance on the IslamicMMLU leaderboard and strong resistance to false premises), and lessons for faith‑sensitive LLM deployments.

By M Waleed Kadous, Amr Elsayed, Abdullah Al Nahas, Ashraf Haress
arXiv Computation and Language
Sep 22

Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model

The paper presents a smartphone‑compatible, retrieval‑augmented language model tailored to Bangladeshi statutory law. By distilling a 9‑billion‑parameter Gemma‑2 teacher into a 2‑billion‑parameter student using supervised fine‑tuning and QLoRA, the authors achieve significant gains in ROUGE‑L and BERTScore on an English benchmark while keeping the model lightweight (1.6 GB) and operable offline on a Pixel 6. The system retrieves from 36,029 statutory passages using a hybrid dense/BM25 approach, and cross‑lingual evaluation shows effective Bangla query handling against an English‑only corpus, with a practicing lawyer rating the responses highly in a pilot study.

By MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan, Nadifa Zaman, Fariha Roushon Florin, Farig Yousuf Sadeque, Saadat Rafid Ahmed