The study evaluated ten chest X‑ray AI classification models on 12 datasets from six countries, focusing on image‑level classification and measuring specificity at 95% sensitivity. MAVL and MedKLIP achieved the highest specificity (≈0.37) but still produced many false positives, with performance dropping on institutional cohorts compared to public benchmarks. No single model excelled across all findings, countries, or lesion types, highlighting limited generalisability.
By Qinmei Xu, Yiheng Li, Wenya Zhao, Zihao Zhao, Chuanjun Xu, Ahmet Gorkem Er, Xianghao Zhan, Brittany Dashevsky, Mohammed Alawad, Marc von der St\"uck, Marvin Gazibari\'c, Daniel Truhn, Shunsuke Shibata, Ryosuke Horino, Mio Ito, Akio Hiwatashi, Suyu Li, Mengya Yang, Ya Liu, Zhiyuan Sun, Changsheng Zhou, Haruka Itakura, Xiao Li, Olivier Gevaert
arXiv:2606. 17710v1 Announce Type: cross Abstract: Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image.
By Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
Med-AR introduces two autoregressive vision‑language models, Med‑AR‑8B and Med‑AR‑2B, pretrained on structured radiology reports, abnormality‑focused text, and region annotations to address long‑tailed chest X‑ray classification. The models outperform existing contrastive, self‑supervised, and supervised encoders—including Med‑CLIP, CheXFound, EVA‑Base, ARK, and BioViL‑T—across PadChest, MIMIC‑CXR, and CheXpert, achieving higher mean AUROC and AUPRC for head, medium, and tail findings and lower excess area under the risk‑coverage curve. Med‑AR also demonstrates improved selective‑prediction performance, with Med‑AR‑8B raising tail‑label mean AUPRC on MIMIC‑CXR from 0.1033 to 0.1441 and Med‑AR‑2B delivering the strongest discrimination on PadChest.
By Janhavi Prabhu, Sahil, Akshay V, Shivam Shukla, Manoj Tadepalli, Preetham Putha
arXiv:2607. 25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases.
By Mateusz Koz{\l}owski
Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image.
The study introduces GazeRIB, a method that uses radiologists’ gaze data to mask unseen regions during training of chest X‑ray models. By comparing panels of three, five, and eight readers against individual readers across multiple architectures, the authors found that panels consistently improved macro F1 scores by 5.33–7.47 percentage points, with three readers capturing most of the benefit. External validation showed panel‑based models also enhanced ROC‑AUC on the SIIM‑FISABIO‑RSNA dataset, though results varied on other cohorts.
By Nattakorn Kittisut, Roy M. Gabriel, Marly van Assen, Mohammadreza Zandehshahvar, Carlo N. De Cecco, Elizabeth A. Krupinski, Ali Adibi
arXiv:2509. 19671v3 Announce Type: replace Abstract: Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare.
By Andrew Wang, Jiashuo Zhang, Michael Oberst
arXiv:2607. 26333v1 Announce Type: cross Abstract: Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment.
By Panagiotis Fytas, Ian Selby, Clemens Karner, Judith Babar, Simon Baker, Jake Beckford, Timothy J. Sadler, Shahab Shahipasand, Arthikkaa Thavakumar, John Li Chen, Alex Sawer, Michael Roberts, Jonathan Weir-McCall, J. H. F. Rudd, Carola-Bibiane Sch\"onlieb, Anna Korhonen, Anna Breger
arXiv:2606. 02276v1 Announce Type: cross Abstract: Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence.
By Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn
The study evaluates four zero‑shot System 1 models on 86 public clinical tasks across 14 domains and 20 languages, measuring performance via AUROC. The best models achieved high AUROC scores—0.94 for medical knowledge, 0.93 for text understanding, 0.83 for clinical judgement from text, 0.89 for image perception, 0.84 for 2D image findings, and 0.69 for volumetric findings—yet performed worse on imaging tasks and varied significantly with language. System 1 models were fast (0.34–0.44 s) and energy‑efficient (24–81 J) but lagged behind clinicians on triage and were outperformed by generative models in some metrics.
By Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben, Fabian Drexel, Jannik L\"ubberstedt, Johannes Moll, Anirudh Narayanan, Frederike Stumpf, Cosmin I. Bercea, Jiawen Wang, Evangelia E. Vassalou, Krischan Braitsch, Sebastian Ziegelmayer, Markus Graf, Marcus R. Makowski, Lisa Adams, Nassir Navab, Michail E. Klontzas, Keno Bressem
arXiv:2609.22281v1 Announce Type: new
Abstract: Foundation models have recently demonstrated strong capabilities across a wide range of medical imaging tasks. However, their performance in structured...
By Benjamin Renoust, Pierre Baudot, Tiffany Foriel, Yousra Haddou, Charles Voyton, Pierre-Henri Siot, Ezequiel Geremia, Danny Francis, Jean-Christophe Brisset, Val\'erie Bourd\`es, Sylvain Bodard, Benoit Huet
The paper introduces selective linkage precision, a metric for evaluating how accurately vision‑language models can match de‑identified chest radiographs to their reports when only certain attributes (such as sex, age, projection, or acquisition year) are available. Experiments on eight public VLMs and three chest‑radiograph datasets show that filtering by these attributes dramatically improves precision—from a baseline of 1.98% to 67.68% with sex, age, and projection, and up to 99.05% when acquisition year is added—while keeping the candidate set manageable. The study demonstrates that this approach can be applied without retraining the models and may guide hospitals in deciding which metadata to release.
By Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn