← Back to all news
arXiv Computation and Language By Ayana Niwa, Miyu Yamada, Kouta Nakayama, Masahiro Kaneko, Hisami Suzuki, Satoshi Sekine

Rethinking the Evaluation of LLM Fact-Checkers

Read the original on arXiv Computation and Language →

The paper introduces JSFactCheckBench, a Japanese benchmark that explicitly defines the three key design decisions for evaluating large language model (LLM) fact-checkers: the questions the claims answer, the conditions under which each claim is judged, and the sources considered as evidence. It contains 1,913 claims derived from LLM answers to questions about misleading Japanese social‑media content, each annotated for factuality at a reference time and for importance to the question, using evidence from preselected Web sources. Experiments with six LLMs reveal that models are less accurate on claims that directly answer the question than on supplementary claims, that accuracy drops when models must locate relevant content in longer or multiple sources, and that all four open models perform worse when the surrounding question and response are included—weaknesses that are only apparent because the benchmark records the conditions behind each label.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Sep 10

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

SWORD is a new benchmark that tests large language models’ ability to reject factually incorrect statements across eight major languages by distorting Wikidata triples. The benchmark reveals that models often perform better on semantically plausible distortions than on random ones, indicating a reliance on distributional familiarity rather than true factual verification. It also shows significant performance drops for East Asian languages, with gaps up to 28 percentage points, highlighting asymmetric multilingual factual reasoning capabilities.

By Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun
llmsbenchmarks
More like this →
arXiv AI
Oct 16

Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard

The paper introduces FACTGUARD, a multi-task workflow that generates evidence‑grounded answerable and unanswerable questions to improve long‑context reasoning in large language models. It presents FACTGUARD‑BENCH, a benchmark of 25,220 examples with context lengths up to 128K, focusing on Lack of Evidence and Misleading Evidence scenarios. Experiments on eleven LLMs show a reliability gap, with the best model achieving only 71.62% accuracy; training on FACTGUARD‑BENCH raises accuracy to 81.17% and enhances reasoned rejection of unanswerable queries.

By Qian-Wen Zhang, Biao Liu, Fang Li, Jie Wang, Lingfeng Qiao, Yifei Yu, Di Yin, Xing Sun
llmsbenchmarkssafety
More like this →
arXiv AI
Sep 10

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

The paper introduces Fact-Ablated Evaluation (FAE), a framework that iteratively removes cited evidence to test whether large language models (LLMs) adjust their fact‑checking predictions accordingly. Experiments reveal that many off‑the‑shelf LLMs rely more on internal knowledge than on the provided evidence. To address this, the authors propose REAL, a training method that uses counterfactual evidence supervision to encourage LLMs to base veracity judgments on evidence, achieving better evidence‑dependent performance across four datasets.

By Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson
llmsfine-tuningbenchmarks
More like this →
arXiv AI
Sep 25

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.

By Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov, Valeriia Kuschenko, Alexander Evseev, Sergey Bolovtsov
llmsbenchmarkssafety
More like this →