Large language model assistants are being deployed to clinicians, but their readiness is judged mainly by benchmark scores that are based on examination questions or curated cases. This study analyzed 127,833 queries from 6,342 healthcare professionals across 35 specialties, finding that most requests were for documentation, administration, and knowledge retrieval, with a significant portion of queries unanswerable. When compared to 58 public benchmarks, the median benchmark lacked documentation requests and only shared 31% of the real task mix, indicating that current benchmarks poorly reflect actual clinical use.
By Krithik Vishwanath, Haitong Lin, Anton Alyakin, Jin Vivian Lee, D. Brock Hewitt, Jie J. Yao, William Robert Small, Hammad A. Khan, Cordelia Orillac, Aakaash Varma, Brandon Ye, Daniel Alexander Alber, Gustavo Stolovitzky, Batia Wiesenfeld, Oded Nov, Wei Wu, Kang Zhang, Yindalon Aphinyanaphongs, Tim Requarth, Eric Karl Oermann, The International Digital Twin Consortium in Healthcare, Medicine
arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.
By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
The paper introduces ICD-Deepresearch, a workflow that combines foundation models for electronic health records (EHR) and language models with medical search and ICD dictionaries to forecast future ICD codes for upcoming clinical encounters. It evaluates candidate code transitions by linking patient evidence, external clinical relations, and exact code semantics within a fixed top‑K budget, using SparseEHR for initial priors, GPT‑5 for complementary forecasts, and a final selection step that validates, deduplicates, and ranks candidates. The method achieves patient‑averaged precision/recall of 24.60/35.09% on MIMIC‑III and 25.14/48.32% on MIMIC‑IV, with physicians rating 51–68% of its retrieved documents as useful, outperforming standalone GPT‑5 web search and Medical Deep Research.
By Junda Wang, Meysam Ghaffari, Akshat Choube, Mohsen Sharifi Renani, Hong Yu, Carlos Morato
arXiv:2607. 25485v1 Announce Type: new Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf.
By Korosh Vatanparvar, Ashutosh Joshi, Maria Xenochristou, Mohammad Abuzar Hashemi, Prasad Kasu, Deepak Bansal, Daniel Lopez-Martinez, Anchal Nema, Ramya Ganesan, Will Kimbrough, Alex Woody, Yadunandana Rao, Dilek Hakkani-Tur, Wilko Schulz-Mahlendorf
arXiv:2505. 14107v5 Announce Type: replace-cross Abstract: The emergence of groundbreaking large language models capable of performing complex reasoning tasks holds significant promise for addressing various scientific challenges, including those arising in complex clinical scenarios.
By Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, Xiaofan Zhang
arXiv:2609.24480v1 Announce Type: cross
Abstract: Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the conv...
By Kalash Shah, Kunal Singh, Snehan J, Shreyas Singh