← Back to all news
arXiv Computation and Language By Kazuma Iwamoto, Kazumasa Omura, Shotaro Ishihara

An Empirical Analysis of Factual Errors in Human-Written Text and Its Application to Factual Error Detection

Read the original on arXiv Computation and Language →

The paper presents an empirical study of factual errors in human-written text, focusing on corrections in newspaper articles to build a taxonomy of common mistakes such as kanji misconversions and unit errors. It evaluates large language models’ ability to detect these errors, finding that even advanced models like GPT‑5.4 achieve only a 52% word‑level F1 score on synthetic data, underscoring the difficulty of the task. The work highlights the gap in research on factual error detection in human writing compared to LLM hallucinations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Aug 27

From Passive Response to Proactive Correction: Enhancing LLM Robustness Against Input Fact Perturbations

The paper introduces DEDUCE, a three‑stage framework that turns large language models into proactive error correctors by detecting input fact errors, devising correction strategies, and delivering reliable answers. It also presents MisFactQA, a dataset of factual errors, and new metrics for robustness evaluation. Experiments on TruthfulQA, FalseQA, and MisFactQA show significant gains in accuracy and error correction across Qwen, LLaMA, and Gemma models.

By Ping Wang, Xiangguo Sun, Bingbing Xu, Guocong Li, Xiaofeng Meng
llmsbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 10

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

SWORD is a new benchmark that tests large language models’ ability to reject factually incorrect statements across eight major languages by distorting Wikidata triples. The benchmark reveals that models often perform better on semantically plausible distortions than on random ones, indicating a reliance on distributional familiarity rather than true factual verification. It also shows significant performance drops for East Asian languages, with gaps up to 28 percentage points, highlighting asymmetric multilingual factual reasoning capabilities.

By Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun
llmsbenchmarks
More like this →