arXiv:2604. 14892v3 Announce Type: replace-cross Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators.
By Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor, Michael Cameron Gramanie, Ismail Kalla, Shabir A. Madhi, Kajal Morar, Marlvin T. Ncube, Haroon Saloojee, Bruce A. Bassett
arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.
By Samiha A. Ismail, Fan X. Chen, Ali Merali
arXiv:2611.00460v1 Announce Type: cross
Abstract: Large language models (LLMs) now answer medical exam questions at or above expert level, but it is unclear whether such scores reflect clinical compe...
By Navapat Nananukul, Mayank Kejriwal
arXiv:2608.22622v1 Announce Type: cross
Abstract: Clinical decision-making relies on identifying relevant patient information to guide diagnosis and treatment, a challenge that is especially difficul...
By Miguel Contreras, Scott Siegel, Subhash Nerella, Jessica Sena, Jiaqing Zhang, Heng Sun, Hruday Tej Akkaladevi, Peiyu Lu, Jordan Rosen, Sumit Kapoor, Sasank Desaraju, Grace R. Thompson, Jacob Purcell, Michael Petrauskis, Philip KW. Hong, Meghan Brennan, Sarah Chrabaszcz, Tierra Smith, Ronnie Ren, Michel S. Kabbash, Ceyhun Haziroglu, Rushi Patel, Gabriel Gomez, Charlotte Chaiklin, Randy Leung, Kenneth N. John, Whitman Wiggins, Philip Kayser, Vincent Bird, Maria Bruzzone, Tyler J. Loftus, Azra Bihorac, Parisa Rashidi
arXiv:2607. 19678v1 Announce Type: cross Abstract: AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a single final answer.
By Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull, Gregory E Dean, Maria Liakata
The paper introduces Clinical System One (CS1), a 4‑billion‑parameter decision model that rapidly produces typed judgments from clinical records. Trained on 12.4 million Q&A pairs from 3.6 million patients, CS1 can answer multiple questions in a single forward pass, providing probability estimates for uncertain cases. In evaluations on 963,408 held‑out patients and 21 external datasets, CS1 outperformed larger open models and, when paired with a language‑model agent, efficiently identified trial‑eligible patients with far fewer tokens.
By Andrew Zhang, Trinh T. L. Vuong, Sophia J. Wagner, Dongmin Bang, Tong Ding, Yuchang Su, Mary Cipperman, Christina S. Chen, Judith Lefkes, Ramin Kahidi, Ming Y. Lu, Caiwei Tian, Tianhao Luo, Jingnan Ma, Rowland Pettit, Long Phi Le, Faisal Mahmood
arXiv:2409. 07314v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows.
By Praveenkumar Kanithi, Cl\'ement Christophe, Marco AF Pimentel, Tathagata Raha, Prateek Munjal, Nada Saadi, Hamza A Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan
arXiv:2606. 28960v1 Announce Type: new Abstract: Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice.
By Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena
arXiv:2505. 14107v5 Announce Type: replace-cross Abstract: The emergence of groundbreaking large language models capable of performing complex reasoning tasks holds significant promise for addressing various scientific challenges, including those arising in complex clinical scenarios.
By Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, Xiaofan Zhang
arXiv:2608. 16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation.
By Yifan Zhang, Rahmatollah Beheshti
arXiv:2609.37788v1 Announce Type: cross
Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, draw...
By Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo
The paper introduces Med Evidence Bench (MEB), a benchmark of 1,536 physician‑constructed items covering guideline knowledge, standardized clinical scenarios, and real‑world laboratory scenarios, each linked to a versioned source‑text span. Across ten large language models, the Evidence Trace Score (ETS) ranged from 0.975 to 0.997 for guideline knowledge, 0.292 to 0.658 for standardized scenarios, and 0.182 to 0.447 for real‑world scenarios, with varying rankings across levels. Automated grading on 1,296 criterion‑level judgments matched the majority vote of three physicians with a macro‑F1 of 0.930, though agreement was lower for evidence application.
By Zhiqun Zhao (Advanced Medical Research Institute and Key Laboratory for Experimental Teratology of the Ministry of Education, Cheeloo College of Medicine, Shandong University, Jinan, China), Xiaodong Yang (Department of Urology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China), Bruce Gu (Shandong Computer Science Center, National Supercomputer Center in Jinan, Jinan, China, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China), Ming Li (Department of Rheumatology and Immunology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China), Yi Fu (Information Network Management Office, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China), Song Li (Department of Medical Oncology, Qilu Hospital, Shandong University, Jinan, China), Qi Mei (Department of Oncology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China), Zhi Liu (Department of Clinical Laboratory, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China), Kai Zhang (Department of Breast Surgery, General Surgery, Qilu Hospital, Shandong University, Jinan, China), Kexue Fu (Shandong Computer Science Center, National Supercomputer Center in Jinan, Jinan, China, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China), Jing Wu (Department of Central Laboratory, Public Health Clinical Center Affiliated to Shandong University, Shandong University, Jinan, China), Xiaofei Lu (Department of Breast Surgery, The First Affiliated Hospital of Shandong First Medical University, Jinan, China), Yule Li (Advanced Medical Research Institute and Key Laboratory for Experimental Teratology of the Ministry of Education, Cheeloo College of Medicine, Shandong University, Jinan, China), Zheng Wang (Department of Urology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan, China), Haiquan Lu (Advanced Medical Research Institute and Key Laboratory for Experimental Teratology of the Ministry of Education, Cheeloo College of Medicine, Shandong University, Jinan, China), Xiaoming Wu (Shandong Computer Science Center, National Supercomputer Center in Jinan, Jinan, China, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China)