MS-Exam-Gen is a reproducible framework that builds a source‑grounded multiple‑choice question benchmark for evaluating large language models on knowledge about multiple sclerosis MRI. The pipeline uses expert‑indexed sources, topic induction, evidence‑grounded MCQ generation, automated quality audits, and consistency checks to produce a 3,058‑item benchmark covering 16 topics and 53 subtopics. Evaluation of 12 LLM endpoints on this benchmark revealed a wide accuracy range (89.7% to 46.9%) and identified items frequently missed by models, while audits showed reduced answer cues and position‑sensitivity in scoring.
arXiv:2609.34024v1 Announce Type: cross
Abstract: Jev is a non-generative "System One" model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and c...
By Alfredo Madrid-Garc\'ia, Beatriz Merino-Barbancho
arXiv:2606. 07721v1 Announce Type: new Abstract: Objectives: Automatic data extraction from free-text radiology reports enables large-scale research, but few studies assessed the performance of large language models (LLMs) on Dutch neuroradiology reports.
By Kaouther Mouheb, Amos Pomp, Antoine Manenti, Romy de Haan, Farog Faghir, Joy Martens, Harro Seelaar, Francesco Mattace-Raso, Meike W. Vernooij, Frank J. Wolters, Stefan Klein, Esther E. Bron
The study evaluates Jev 1.13, a non‑generative model that selects from predefined answer options, on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena‑MCQ, and the NEJM Case Challenges. Jev’s top‑1 accuracy matches GPT‑6 Sol with medium reasoning on PubMedQA but falls behind on MetaMedQA, DiagnosisArena‑MCQ, and NEJM cases. While Jev shows strong calibration on MetaMedQA and is fast and inexpensive, its performance on examination and complex diagnostic tasks is substantially lower, indicating the need for task‑specific validation before clinical deployment.
By Alfredo Madrid-Garc\'ia, Beatriz Merino-Barbancho
An integrated, web‑based platform has been developed to process diffusion‑weighted imaging (DWI) from MR‑guided linear accelerators and provide structured, literature‑grounded clinical interpretations. The system combines a deep‑learning pipeline for distortion correction, denoising, and IVIM/ADC fitting with a retrieval‑augmented generation (RAG) agent that references a curated knowledge base and traces each statement to its source. Independent expert ratings of nine glioblastoma cases showed high scores for clinical reasoning, citation quality, and overall utility, with a mean rating of 4.65 out of 5.
By Yunxiang Li, Yan Dai, Yen-Peng Liao, Jie Deng, Jill B De Vis, You Zhang
arXiv:2610.15239v1 Announce Type: new
Abstract: Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinic...
By Anurag Sharma, Aditya Sharma, Sapna Baalwan
arXiv:2608.22323v1 Announce Type: new
Abstract: The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus...
By Lai Wei, Yuchao Chen, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Bangting Wang, Wei Chen, Xiang Bai
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or...
arXiv:2606. 15735v1 Announce Type: cross Abstract: Discharge summaries are crucial clinical documents containing the context of a patient's overall hospital stay, and are routinely reviewed by medical experts for patient readmission, ongoing care, and diagnostic decision-making.
By Jiyoun Kim, Muhan Yeo, Eunhye Jang, Jeewon Yang, Hangyul Yoon, Su Ji Lee, Hee Jo Han, Hee-Jae Jung, Doyun Kwon, Jun young Lee, Jaehun Lee, Jung-Oh Lee, Sunjun Kweon, Jong Hak Moon, Daseul Kim, Minjae Cho, Edward Choi
arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.
By Samiha A. Ismail, Fan X. Chen, Ali Merali
arXiv:2604. 14892v3 Announce Type: replace-cross Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators.
By Amy Rouillard, Sitwala Mundia, Linda Camara, Ziyaad Dangor, Michael Cameron Gramanie, Ismail Kalla, Shabir A. Madhi, Kajal Morar, Marlvin T. Ncube, Haroon Saloojee, Bruce A. Bassett
arXiv:2609.32807v2 Announce Type: replace
Abstract: Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustnes...
By Chaitali Deb Purkayastha, Bharath Kumar Bolla, Vishnu Surya Reddy Nandi