arXiv Machine Learning September 21, 2026 By Mushir Akhtar, M. Tanveer, Mohd. Arshad

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

Read the original on arXiv Machine Learning →

The study evaluates the robustness of medical vision‑language models for tuberculosis screening on chest X‑rays by testing them across multiple datasets, prompts, and evaluation settings. Three specialized models (BioMedCLIP, CheXficient, MedSigLIP) and a general OpenCLIP model were audited on 12,200 images, producing 244,000 model–image–prompt scores. Results show that no model consistently outperforms others across all cohorts and reliability criteria, with prompt changes and control group composition significantly affecting AUROC, and that high training‑set performance does not reliably transfer to external cohorts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 17

Vision-language models for chest radiography do not always need the image

arXiv:2606. 17710v1 Announce Type: cross Abstract: Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image.

By Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh

llms multimodal benchmarks

arXiv AI

Jul 31

Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

arXiv:2607. 25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases.

By Mateusz Koz{\l}owski

llms multimodal benchmarks

Hugging Face Trending Papers

Jun 29

SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image.

llms multimodal benchmarks

arXiv Machine Learning

Jun 29

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

arXiv:2509. 19671v3 Announce Type: replace Abstract: Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare.

By Andrew Wang, Jiashuo Zhang, Michael Oberst

computer-vision benchmarks

arXiv Machine Learning

Jul 30

Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance

arXiv:2607. 26333v1 Announce Type: cross Abstract: Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment.

By Panagiotis Fytas, Ian Selby, Clemens Karner, Judith Babar, Simon Baker, Jake Beckford, Timothy J. Sadler, Shahab Shahipasand, Arthikkaa Thavakumar, John Li Chen, Alex Sawer, Michael Roberts, Jonathan Weir-McCall, J. H. F. Rudd, Carola-Bibiane Sch\"onlieb, Anna Korhonen, Anna Breger

llms fine-tuning multimodal safety

arXiv AI

Jun 2

Cross-modal linkage risk in clinical vision-language models

arXiv:2606. 02276v1 Announce Type: cross Abstract: Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence.

By Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung, Daniel Truhn

llms rag fine-tuning multimodal benchmarks safety

One email a morning, machine-written

Related stories

Vision-language models for chest radiography do not always need the image

Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance

Cross-modal linkage risk in clinical vision-language models