arXiv Machine Learning
Oct 7

Cite What You Explore: Budget-Aware LLM Reasoning over Medical KGs with Verifiable Evidence

The paper introduces BAR, a Budget‑Aware LLM Reasoning framework that enhances post‑discharge risk prediction by integrating external medical knowledge graphs (KGs). BAR refines KGs into disease‑specific evidence graphs with support scores and provenance, then uses an LLM to plan, navigate, and verify evidence within a patient‑specific budget. Experiments on MIMIC‑III and MIMIC‑IV across eight diseases show BAR improves AUPRC by 3.4 points, raises citation precision from 59.8% to 77.9%, and uses only 62‑65% of the allotted budget.

By Chen Chen, Dongjie Wang, Mei Liu, Zijun Yao
arXiv AI
Sep 25

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a continuously maintainable benchmark for evaluating large language models (LLMs) in electronic health record (EHR) information retrieval. It presents a scalable framework that automatically generates question–answer pairs from longitudinal EHR notes, validated by nineteen clinicians. The benchmark allows assessment of multiple inference strategies and highlights that state‑of‑the‑art LLMs often miss clinically important information, especially when synthesis across documents is required.

By Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani, Philip Chung, Kevin R Keet, Kameron C. Black, Andrea T. Fisher, Sarita Khemani, Jerry Liu, Stephen Ma, Saloni K. Maharaj, Rita M. Pandya, Eduardo Perez-Guerrero, Priyanka Pillai, Lisa Shieh, David J. H. Wu, James Xie, James C. McAvoy, Teresa Nguyen, Jessica Tran, Lucy Yin, Bridget Lin, Alison Callahan, Jason A. Fries, Nigam H. Shah, Emily Alsentzer
Hugging Face Trending Papers
Sep 24

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a scalable framework that automatically creates question–answer pairs from longitudinal electronic health record notes, validated by nineteen clinicians. It offers a continuously maintainable benchmark for evaluating large language models in clinical settings, addressing limitations of manual, costly, and quickly outdated existing benchmarks. Experiments across nine LLMs and five inference strategies reveal that even state‑of‑the‑art systems often miss clinically important information, especially for synthesis‑heavy queries.

arXiv AI
Aug 11

CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR

arXiv:2608. 07796v1 Announce Type: new Abstract: Large language models perform strongly on medical knowledge benchmarks, but reliable clinical deployment requires agents to conduct defensible investigations over heterogeneous, longitudinal records: determining what evidence is needed, retrieving and reconciling structured and free-text data, grounding conclusions in verifiable evidence, and deferring cases that cannot be resolved reliably.

By Veronica Chatrath, Bryan Zhu, George Pu, Jingxuan Fan, Apaar Shanker, Varun Ursekar, Anahita Sharma, Jason Qin, Keqi Han, Soham Dinesh Tiwari, Soham Dan, Vijay Kalmath, Yuan Li, Daniel Yue Zhang, Chenguang Wang, Zainab Doctor, Zhijun Yin, Nigam H. Shah, Yuan Xue
arXiv AI
Aug 28

Structured Evidence Routing for Incident Risk Prediction from Multimodal Longitudinal EHRs

The paper introduces structured evidence routing for incident risk prediction using multimodal longitudinal electronic health records (EHRs). It proposes a router‑predictor‑reviewer workflow that condenses full patient records into compact summaries and targeted evidence slices, enabling a predictor to generate evidence‑linked risk assessments that a reviewer can critique. Experiments on five one‑year incident diagnosis tasks show that the method matches the AUROC of established supervised EHRSHOT baselines and remains competitive on AUPRC, while providing a patient‑specific evidence trail.

By Animesh Agarwal, Meysam Ghaffari, Nina Fatehi, Carlos Morato