← Back to all news
arXiv AI By Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

Read the original on arXiv AI →

The paper introduces Fact-Ablated Evaluation (FAE), a framework that iteratively removes cited evidence to test whether large language models (LLMs) adjust their fact‑checking predictions accordingly. Experiments reveal that many off‑the‑shelf LLMs rely more on internal knowledge than on the provided evidence. To address this, the authors propose REAL, a training method that uses counterfactual evidence supervision to encourage LLMs to base veracity judgments on evidence, achieving better evidence‑dependent performance across four datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Aug 28

AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification

AEScorer is an agentic evidence‑grounded framework designed for graded factuality verification, addressing the limitation of binary judgments in current methods. It operates in two stages: first, it gathers and refines external evidence through agentic search; second, it predicts a scalar factuality score to capture nuanced differences in correctness. The authors also introduce GradedVeriBench, a benchmark covering general and multi‑hop question answering, and demonstrate that AEScorer outperforms existing approaches on this new benchmark.

By Hui Huang, Muyun Yang, Yuki Arase
llmsagentsnlpbenchmarks
More like this →
arXiv AI
Oct 16

Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard

The paper introduces FACTGUARD, a multi-task workflow that generates evidence‑grounded answerable and unanswerable questions to improve long‑context reasoning in large language models. It presents FACTGUARD‑BENCH, a benchmark of 25,220 examples with context lengths up to 128K, focusing on Lack of Evidence and Misleading Evidence scenarios. Experiments on eleven LLMs show a reliability gap, with the best model achieving only 71.62% accuracy; training on FACTGUARD‑BENCH raises accuracy to 81.17% and enhances reasoned rejection of unanswerable queries.

By Qian-Wen Zhang, Biao Liu, Fang Li, Jie Wang, Lingfeng Qiao, Yifei Yu, Di Yin, Xing Sun
llmsbenchmarkssafety
More like this →
arXiv AI
Sep 10

Evidence-Aligned Entity Verification for Hallucination Detection in Retrieval-Augmented Generation

The paper introduces Evidence-Aligned Entity Verification (EAEV), a method for detecting entity-level hallucinations in retrieval-augmented generation (RAG). EAEV aligns generated entities with retrieved evidence across three dimensions and uses counterfactual stability analysis to maintain robust alignments when evidence changes. Experiments on multiple RAG benchmarks show that EAEV consistently outperforms existing hallucination detection methods and generalizes well.

By Runsong Jia, Zhen Fang, Mengjia Wu, Jie Lu, Yi Zhang
llmsragbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 23

ARAFA: An LLM-Generated Arabic Fact-Checking Dataset

A new large-scale Arabic fact‑checking dataset called Arafa has been created using an automated pipeline that generates claims from Arabic Wikipedia, mutates them into counterfactuals, and validates them against supporting or refuting evidence. The dataset contains 181,976 claim‑evidence pairs labeled as supported, refuted, or not enough information, and human evaluation shows high inter‑annotator agreement and strong validation accuracy. Fine‑tuned transformer models on Arafa achieve a Macro F1‑score of 77%, demonstrating its usefulness for Arabic fact‑checking tasks.

By Christophe Khalil, Shady Elbassuoni, Rida Assaf
llmsnlpfine-tuning
More like this →