The study evaluates counterfactual bias in ten open‑source large language models (LLMs) for pediatric Emergency Severity Index (ESI) prediction. By creating paired clinical vignettes that differ only in demographic or socioeconomic variables, the authors measure shifts in acuity assignment, finding that counterfactual sensitivity varies widely across model families and sizes. A fine‑tuned Qwen2.5‑7B model exhibited the lowest sensitivity, while larger or medical‑domain models sometimes showed greater shifts, highlighting the need for fairness assessment before clinical deployment.
By Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang
arXiv:2608.29582v1 Announce Type: cross
Abstract: Current evaluations of large language models (LLMs) primarily focus on factual knowledge retrieval, overlooking the fundamental challenge of navigati...
By Yi Yu, Bo Wang, Chong Feng, Ge Shi, Xia Liu, Ziyi Yang, Xuewen Shi
arXiv:2606. 05970v1 Announce Type: cross Abstract: Large language models are increasingly used for structured extraction from clinical free-text notes, but the sensitivity of their output to upstream configuration choices is less understood than their accuracy on fixed benchmarks.
By Martin Murin
Background: Disease severity is a multidimensional construct difficult to capture with rule-based approaches in Electronic Healthcare Records (EHR). Agentic large language model (LLM) systems could synthesise clinical evidence and reason over EHRs, but remain unevaluated for this task.
The paper introduces ClinGeX, a large‑language‑model agent that uses 105 curated executable skills to automate clinical genomic analysis. In benchmark tests, ClinGeX achieved 71.7% macro‑averaged task‑level concordance, outperforming the best comparator at 45.7%. Human‑AI collaboration studies showed ClinGeX significantly improved case‑level concordance across variant analysis, rare disease interpretation, and oncology decision‑making, while also reducing interpretation time.
By Qiaoyu Zheng, Weike Zhao, Yanjie Fan, Ye Cao, Mengqi Jiang, Xiao Zhou, Xiaoman Zhang, Pengcheng Qiu, Yongguo Yu, Yanfeng Wang, Ya Zhang, Chaoyi Wu, Weidi Xie
The paper introduces VERGE, a verification-enhanced refinement workflow that extracts six red‑flag symptoms and family‑history risk status for early‑onset colorectal cancer from free‑text clinical notes. VERGE uses retrieval‑augmented generation followed by a bounded verification‑refinement cycle that checks textual grounding and clinical validity, correcting claims until resolved or escalating to human review. In evaluation on 4,033 clinician‑labeled note‑finding pairs, VERGE improved precision from 0.764 to 0.849 and MCC from 0.681 to 0.730 compared to a single‑agent baseline, while requiring human review for only 1.5 % of claims.
By Nikkie Hooman, Monarch Nigam, Amy E. Hughes, Rasmi G. Nair, Mehak Gupta
The study evaluates large language models (LLMs) on sequential emergency department triage, where acuity labels are predicted from progressively longer nurse‑patient conversations. Six LLMs were tested at five checkpoints on simulated and physician‑authored dialogues, showing a decline from moderate‑to‑substantial agreement on full records to only fair‑to‑moderate agreement at each checkpoint. The models consistently anchor on chief complaint exchanges and fail to integrate later evidence, yielding low agreement with clinicians (QWK 0.295 vs. 0.887‑0.929) and concentrating predictions on ESI‑2 and ESI‑3.
whyItMatters":"The findings reveal that LLMs, despite strong offline performance, cannot reliably handle the sequential nature of real‑time triage, highlighting a critical gap for safe deployment in emergency settings."
By Dipankar Srirag, Haokai Zhao, Ashutosh Kumar, Eleanor Hopper, Michael Dalton, Quoc Dung Nguyen, Aditya Joshi, Salil S. Kanhere, Padmanesan Narasimhan
The paper introduces AI Consilium, a benchmark for evaluating large language models (LLMs) on providing clinically grounded, coherent, actionable, and safe second opinions in multidisciplinary settings. Using 300 NEJM–MedQA cases, the study compares eight single-model pipelines that generate role-specific opinions and interactive-consilium reports under free-form and structured conditions. Results show that structured final reports generally score higher, especially in management, and that GPT‑5.6‑Sol outperforms other models across all evaluated components, demonstrating the benchmark’s ability to differentiate model capabilities and the impact of response structure.
By Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev, Mikhail Yu. Valkov, Valeria Yu. Chernina, Natalia Gombolevskaya, Ayyaz Hussain, Vladimir M. Agafonov, Victor A. Gombolevskiy, Aleksandr V. Nesterov, Iaroslav R. Bespalov, Ivan V. Oseledets, Dmitry V. Dylov
arXiv:2610.20512v1 Announce Type: new
Abstract: Graph-based analysis of electronic health records (EHRs) yields disease-trajectory catalogs, but structural support does not establish semantic novelty...
By Jin-Jia Zheng, Vincent S. Tseng
arXiv:2604.26766v2 Announce Type: replace-cross
Abstract: Accurate and consistent Emergency Severity Index (ESI) assignment remains a persistent challenge in emergency departments, where highly varia...
By Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang
arXiv:2607. 12886v1 Announce Type: new Abstract: Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields.
By Cameron Cagan, Pedram Fard, Jiazi Tian, Jingya Cheng, Shawn N. Murphy, Hossein Estiri
arXiv:2608. 07299v1 Announce Type: cross Abstract: Radiology reports describe clinical observations but do not specify executable segmentation targets.
By Chengyi Peng, Haoyu Yang, Meixing Shi, Yuxiang Cai, Yankai Jiang