arXiv Computation and Language

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

The paper investigates why retrieval‑based open‑ended evaluation fails in medical fact verification. By creating two detailed taxonomies—one for retrieval‑stage errors across five quality dimensions and another for verifier‑reasoning errors across six steps—the authors automatically label evidence quality and reasoning errors using an LLM‑as‑Judge pipeline. Their large‑scale stress tests across multiple retrieval methods and verifier models show that increasing model size, reasoning effort, source breadth, or medical fine‑tuning does not eliminate these failure modes, indicating fundamental limits of the retrieve‑then‑verify paradigm in open‑ended medical contexts.

arXiv Computation and Language
Aug 24

MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation

MedRAGChecker is a claim-level verification framework designed for biomedical retrieval‑augmented generation (RAG). It decomposes generated answers into atomic claims and assesses each claim’s support by combining evidence‑grounded natural language inference with biomedical knowledge‑graph consistency signals. The aggregated claim decisions provide diagnostics that distinguish retrieval and generation failures, such as faithfulness, under‑evidence, contradiction, and safety‑critical errors, and the system is distilled into compact models for scalable evaluation.

By Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang
arXiv AI
Aug 28

MedFabric: Gold Evidence Hides the Difficulty of Word-Level Medical Fabrication Detection

MedFabric is a new benchmark for detecting word‑level medical fabrications, comprising 646 fabricated statements each paired with a ground‑truth passage that shares the same LLM authorship and nearly identical wording. The study shows that current detectors perform poorly—expert clinicians achieve only 53.3% macro‑F1 and no detector family surpasses 60% without gold evidence—highlighting that detection hinges on evidence correctness rather than subtlety of fabrication. The authors demonstrate that a retrieval‑confidence gate can substantially improve performance, raising macro‑F1 from 61% to 74%.

By Tung Sum Thomas Kwok, Qian Qian, Xiaofeng Lin, Dongxu Zhang, Jun Han, Zhichao Yang, Davin Hill, Tamer Soliman, Sanjit Singh Batra, Robert Tillman, Guang Cheng
arXiv AI
Jun 2

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

arXiv:2603. 05308v3 Announce Type: replace-cross Abstract: Assessing whether an article supports an assertion is essential for hallucination detection and claim verification.

By Qiao Jin, Yin Fang, Lauren He, Yifan Yang, Guangzhi Xiong, Zhizheng Wang, Nicholas Wan, Joey Chan, Donald C. Comeau, Robert Leaman, Charalampos S. Floudas, Aidong Zhang, Michael F. Chiang, Yifan Peng, Zhiyong Lu
arXiv AI
Aug 26

Generating Biomedical Fact-Checking Reports with RL-Enhanced Agentic Search

The paper introduces BioCheck Agent, an LLM-based system that generates structured biomedical fact‑checking reports using agentic search and a reinforcement‑learning framework called EG‑GRPO. Unlike prior methods that output only supported or refuted labels, BioCheck Agent synthesizes conclusions with retrieved evidence from PubMed, employing advanced Boolean search operators. Experiments show that, compared to the base Qwen3.5‑4B model, BioCheck Agent improves label prediction accuracy on SciFact by 9.95 %, raises evidence quality by 3.7 %, and reduces hallucinations by 19.63 %.

By Jiongxiao Wang, Dingli Ma, Chaoqun Ni
arXiv AI
Sep 16

CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine

The paper introduces CLEAR, an agentic framework designed to improve the reliability of large language models (LLMs) in medical contexts by adjudicating evidence from multiple sources. CLEAR generates candidate answers from three distinct pathways—parametric knowledge, locally curated corpora, and dynamically retrieved evidence—and then uses an aggregation verifier to evaluate agreement and conflict among these sources. An adjudication module decides whether to preserve or revise conclusions, employing override-guard and challenge-audit mechanisms, and initiates targeted follow-up searches when conflicts remain unresolved.

By Shuai Wang, Yize Zhao, Qingyu Chen
arXiv Computation and Language
Aug 27

When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification

The paper investigates evidence generation for biomedical claim verification, evaluating various large language models and retrieval strategies on the CARE-XAI benchmark. It finds that fine‑tuned LLMs excel at producing evidence, while biomedical classifiers still lead in verdict‑only prediction. PubMed retrieval helps on PubMed‑aligned datasets but can mislead on broader public‑health claims, prompting the authors to propose Bio‑GRACE, a diagnostic that normalizes gold references to assess retrieval utility.

By Pritam Deka, Prabhjot Singh
arXiv AI
Sep 2

Medical Causal Hypothesis Verification with Large Language Models

The paper "Medical Causal Hypothesis Verification with Large Language Models" reports a small-scale study evaluating eight LLMs on 17 medical causal hypotheses. The authors introduce an evaluation framework and annotate 1,067 evidence points across six criteria, using nine metrics to assess performance. Results show that while LLMs have strong recall, they frequently fail to provide valid scientific articles, evidence, or reject unsupported hypotheses, revealing a critical limitation for their use in healthcare.

By Safiyyah Ahmed, Abrar Ansari, Md Aminul Islam, Elena Zheleva
arXiv AI
Oct 15

Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification

The paper introduces a neurosymbolic verification framework that audits the internal consistency of vision-language models (VLMs) when generating radiology reports. By autoformalizing free‑text findings into structured propositional evidence and using an SMT solver (Z3) alongside a clinical knowledge base, the system determines whether each diagnostic claim is entailed, hallucinated, or omitted. Evaluation on seven VLMs across five chest X‑ray benchmarks reveals distinct reasoning failure modes—such as conservative observation and stochastic hallucination—that traditional metrics miss, and shows that enforcing solver‑backed entailment can eliminate unsupported hallucinations and improve diagnostic soundness and precision.

By Vikash Singh, Debargha Ganguly, Haotian Yu, Chengwei Zhou, Prerna Singh, Brandon Lee, Vipin Chaudhary, Gourav Datta