arXiv Computation and Language
Kurate: Scalable Scientific Quality Analysis
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Sep 2
Medical Causal Hypothesis Verification with Large Language Models
The paper "Medical Causal Hypothesis Verification with Large Language Models" reports a small-scale study evaluating eight LLMs on 17 medical causal hypotheses. The authors introduce an evaluation framework and annotate 1,067 evidence points across six criteria, using nine metrics to assess performance. Results show that while LLMs have strong recall, they frequently fail to provide valid scientific articles, evidence, or reject unsupported hypotheses, revealing a critical limitation for their use in healthcare.
arXiv AI
Jun 19
Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstracts
arXiv:2606. 19345v1 Announce Type: cross Abstract: The rapid increase in scientific publications leads to the fact that manual study screening in systematic literature reviews (SLRs) is increasingly resource consuming, inefficient, and inconsistent.
arXiv AI
Jul 21
Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries
arXiv:2607. 16989v1 Announce Type: cross Abstract: Introduction.
arXiv AI
Jun 2
Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution
arXiv:2603. 05308v3 Announce Type: replace-cross Abstract: Assessing whether an article supports an assertion is essential for hallucination detection and claim verification.
arXiv AI
Aug 28
MedFabric: Gold Evidence Hides the Difficulty of Word-Level Medical Fabrication Detection
MedFabric is a new benchmark for detecting word‑level medical fabrications, comprising 646 fabricated statements each paired with a ground‑truth passage that shares the same LLM authorship and nearly identical wording. The study shows that current detectors perform poorly—expert clinicians achieve only 53.3% macro‑F1 and no detector family surpasses 60% without gold evidence—highlighting that detection hinges on evidence correctness rather than subtlety of fabrication. The authors demonstrate that a retrieval‑confidence gate can substantially improve performance, raising macro‑F1 from 61% to 74%.
arXiv AI
Aug 6
Zero-shot reasoning for simulating scholarly peer-review
arXiv:2510. 02027v2 Announce Type: replace Abstract: Scholarly publishing requires scalable scrutiny supported by auditable evidence.