arXiv Machine Learning August 11, 2026 By Pengyang Yu, Yiou Wang, Zhongping Dong, Sahraoui Dhelim, Chun-Mei Feng, M. Tahar Kechadi

Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions

Read the original on arXiv Machine Learning →

arXiv:2608. 07550v1 Announce Type: cross Abstract: Vision-language models return structured chest-radiograph findings through interfaces exposing no confidence score, so a receiving institution cannot read off how far to trust an individual judgment.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 17

Vision-language models for chest radiography do not always need the image

arXiv:2606. 17710v1 Announce Type: cross Abstract: Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image.

By Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh

llms multimodal benchmarks

arXiv Computation and Language

Aug 21

Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models

arXiv:2606. 15910v2 Announce Type: replace Abstract: A vision-language model can answer a question about a chest radiograph or a pathology slide fluently and confidently while barely using the image, relying instead on language priors.

By Reza Khanmohammadi, Kundan Thind, Mohammad M. Ghassemi

llms agents multimodal

arXiv AI

Jul 31

Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

arXiv:2607. 25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases.

By Mateusz Koz{\l}owski

llms multimodal benchmarks

arXiv AI

Sep 17

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

The study examines how differences in radiologists’ reporting styles—such as terminology, shorthand, formatting, and detail—affect the evaluation of AI-generated chest X‑ray reports. By quantifying the sensitivity of common metrics to these variations, the authors show that changes in reference reports can shift model rankings. They introduce a taxonomy of reporting variations and a rewriting method, ReRef, that preserves clinical meaning while altering style, and release a validated dataset of paired reference reports to aid future research.

By Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny, Nitya M. Bhalla, Fatma Uyar Morency, Pradeep Ravikumar, Zachary C. Lipton, Michael Oberst

arXiv Machine Learning

Sep 21

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

The study evaluates the robustness of medical vision‑language models for tuberculosis screening on chest X‑rays by testing them across multiple datasets, prompts, and evaluation settings. Three specialized models (BioMedCLIP, CheXficient, MedSigLIP) and a general OpenCLIP model were audited on 12,200 images, producing 244,000 model–image–prompt scores. Results show that no model consistently outperforms others across all cohorts and reliability criteria, with prompt changes and control group composition significantly affecting AUROC, and that high training‑set performance does not reliably transfer to external cohorts.

By Mushir Akhtar, M. Tanveer, Mohd. Arshad

llms multimodal benchmarks

arXiv Computer Vision

Sep 22

Authority-Preserving Evaluation of Medical Vision-Language Assistants

arXiv:2609.22302v1 Announce Type: new Abstract: Medical vision-language models can propose how urgently a skin lesion should be reviewed, but the local service retains authority to accept or replace...

By Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong, Roger Wattenhofer

llms multimodal benchmarks safety

One email a morning, machine-written

Related stories

Vision-language models for chest radiography do not always need the image

Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models

Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

Authority-Preserving Evaluation of Medical Vision-Language Assistants