arXiv AI

Model Retirement Creates Reproducibility Risk in Biomedical AI Publications

The study examined biomedical research articles from 2022 to March 2026 that employed large language models (LLMs). It found that 42% of the most frequently used models were already retired or scheduled to retire within two years of publication, with a median retirement interval of 538 days. This high rate of model deprecation threatens the reproducibility of biomedical AI research.

arXiv Computation and Language
Sep 11

The widening evaluation gap in medical large language model research 2023 to 2026

The article examines how medical research lags behind the rapid evolution of large language models (LLMs). From January 2023 to June 2026, PubMed records in fourteen clinical domains grew 45‑fold, yet only 2.5 % employed randomized, controlled, or prospective designs. The evaluation gap widened from 1.33 to 6.08 quarters, with randomized trials assessing models that were on average 4.6 quarters older than other studies, and 62 % of such trials evaluated discontinued model families.

By Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif
arXiv AI
Sep 7

Hakken: Predicting future discoveries to fill the gaps in today's knowledge

Hakken is a domain‑agnostic system that predicts and explains future scientific discoveries by combining transformer‑based models trained on temporal knowledge graphs with large language model semantic knowledge. It identifies novel relationships between scientific concepts that extend beyond the deductive hull of existing knowledge and provides explanations to help scientists assess these predictions. In the biomedical domain, Hakken set a new benchmark for time‑aware multi‑label relation prediction, generated 1.5 million high‑confidence hypotheses about aging, and experimentally confirmed two predictions that revealed previously undocumented interactions relevant to drug discovery.

By Tarek R. Besold, Uchenna Akujuobi, Pablo Sanchez, Alessandra Toniato, Kana Maruyama, Jihun Choi, Samy Badreddine, Frederick Gifford, Daniel Evans-Yamamoto, Sucheendra K. Palaniappan, Miquel Ferrer, Kae Nagano, Iris Rossell, Tom Joy, Hatem ElShazly, Chrysa Iliopoulou, Christoph Wehner, Thiviyan Thanapalasingam, Susana Nunes, Pedro G. Cotovio, Peter Wurman, Peter Stone, Hiroaki Kitano, Michael Spranger
arXiv AI
Jun 2

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

arXiv:2603. 05308v3 Announce Type: replace-cross Abstract: Assessing whether an article supports an assertion is essential for hallucination detection and claim verification.

By Qiao Jin, Yin Fang, Lauren He, Yifan Yang, Guangzhi Xiong, Zhizheng Wang, Nicholas Wan, Joey Chan, Donald C. Comeau, Robert Leaman, Charalampos S. Floudas, Aidong Zhang, Michael F. Chiang, Yifan Peng, Zhiyong Lu
arXiv Computation and Language
Aug 24

MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation

MedRAGChecker is a claim-level verification framework designed for biomedical retrieval‑augmented generation (RAG). It decomposes generated answers into atomic claims and assesses each claim’s support by combining evidence‑grounded natural language inference with biomedical knowledge‑graph consistency signals. The aggregated claim decisions provide diagnostics that distinguish retrieval and generation failures, such as faithfulness, under‑evidence, contradiction, and safety‑critical errors, and the system is distilled into compact models for scalable evaluation.

By Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang