arXiv Computation and Language By Han Chen, Yingrui Li

What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs

Read the original on arXiv Computation and Language →

The paper introduces a joint fact‑verification score that evaluates both answers and the evidence submitted with them. On the FEVEROUS dataset, replacing the DCUF evidence with UnifEE evidence improves the strict score by about 9.6 percentage points, while answer accuracy rises only 1.96 points. The study also shows that increasing context length for large language models yields modest evidence‑gain improvements, and that detailed answer‑evidence analyses uncover patterns missed by aggregate metrics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Oct 31

When the Same Prompt Gives Different Answers (and the Same Answer Isn't Always Right): A Taxonomy of Failure Mechanisms, Testing, and Mitigation Strategies for LLM-Assisted Evidence Synthesis Read-and-Evaluate Tasks

The paper introduces TRAILS, a taxonomy of 38 mechanisms that can compromise the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening, data extraction, risk‑of‑bias assessment, and critical appraisal. It explains how these mechanisms can cause inconsistent or hidden failures across repeated LLM runs and outlines deployment settings, testing approaches, and mitigation strategies for each. The authors argue that repeatability or accuracy alone does not guarantee trustworthy judgments, emphasizing the need for human evaluation to verify that decisions are evidence‑based, free from external influence, and reproducible.

By Carole Lunny, Ian Goodman, Stefano Berrone, Connor TA Brenna, Emma K Reid, Ivan Perez-Neri, Shailesh Kolekar, Nityanand Jain, Ashwin Kamath, Lucas HCC Santos, Yulia Otmakhova, Heba Hussein, Diana M Cespedes, Gioia Dahdal, Salmaan Kanji
arXiv Computation and Language
Nov 4

Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy Distillation

The paper studies how on‑policy distillation (OPD) can improve answer accuracy while reducing a model’s reliance on supporting evidence. Through matched document interventions, the authors identify an "evidence bypass" phenomenon where a student model keeps an originally correct answer even after the supporting document is changed. They propose Evidence‑Paired OPD (EP‑OPD), which uses matched documents and an answer‑level margin to recover most of the counterfactual accuracy lost by standard OPD while preserving much of its original‑document accuracy gains.

By Ruoran Xu, Yilun Zhao, Ziyu Chen, Xiaoqiang kang, Arman Cohan
arXiv AI
Oct 27

Stop or Clarify? Experience-Conditioned Active Fact Verification

The paper introduces SEAC, a method that augments a fixed fact verifier with a lightweight evidence‑acquisition policy to decide when to stop or request more evidence. SEAC evaluates each claim by retrieving evidence, running the verifier, and estimating a clarification‑utility score that balances the benefit of additional evidence against its cost. Experiments on SciFact, LIAR, and FEVER show that SEAC improves mean accuracy and macro‑F1 compared to existing fact‑verification and active‑retrieval baselines.

By Yuhang Dong, Jinghong Li, Xinyue Wang, Yuanning Cui