← Back to all news
arXiv Machine Learning By Aida Usmanova, Zangir Iklassov, Markus Leippold, Ricardo Usbeck

How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation

Read the original on arXiv Machine Learning →

The paper evaluates the robustness of automated fact‑checking systems by cross‑benchmarking nine models—including random baselines, fine‑tuned transformers, zero‑shot LLMs, and top AVeriTeC 2025 systems—across four datasets from scientific, open‑web, and climate domains. It finds that fine‑tuned models outperform zero‑shot LLMs on ClimateCheck, that system rankings vary strongly with domain and metric, and that replacing retrieved evidence with gold annotations boosts veracity accuracy by 14–22 points, underscoring retrieval as the main bottleneck. The authors provide code, pre‑processed datasets, and results to enable reproducible research.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 14

R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration

R2VC is a modular fact‑checking system that separates retrieval, reasoning, verification, and confidence calibration. It uses hybrid sparse‑plus‑dense Wikipedia retrieval, a fine‑tuned generator for structured verdicts, an NLI cross‑encoder for selecting evidence‑based candidates, and a lightweight calibrator for confidence and abstention. On the FEVER benchmark, R2VC improves accuracy by 13.74% over a baseline and shows that verifier‑based candidate selection and calibration are key contributors to performance.

By Dhruv Dixit, Paritosh Pandey
llmsreinforcement-learningfine-tuningbenchmarks
More like this →
arXiv AI
Sep 10

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

The paper introduces Fact-Ablated Evaluation (FAE), a framework that iteratively removes cited evidence to test whether large language models (LLMs) adjust their fact‑checking predictions accordingly. Experiments reveal that many off‑the‑shelf LLMs rely more on internal knowledge than on the provided evidence. To address this, the authors propose REAL, a training method that uses counterfactual evidence supervision to encourage LLMs to base veracity judgments on evidence, achieving better evidence‑dependent performance across four datasets.

By Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson
llmsfine-tuningbenchmarks
More like this →