arXiv AI
Aug 18
MIRROR: Multimodal Intelligent Radiology Reasoning and Observation Reporter
arXiv:2608. 16709v1 Announce Type: cross Abstract: A radiologist reading a model's output faces two problems.
From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks
The study investigates when adding zero‑shot pseudo‑labels from vision‑language models (VLMs) to small expert‑labelled medical imaging datasets ceases to improve, and may even degrade, classifier performance. Across three benchmarks—PatchCamelyon, ISIC 2019, and NIH ChestX‑ray14—the crossover point occurs after roughly 20–50, 100, and 250–500 expert labels, respectively, with additional pseudo‑labels reducing macro‑AUC by up to 0.14 points. The analysis shows that the VLM’s labeling quality, rather than the student model size, limits performance, and a simple bias‑variance rule can predict the benefit of pseudo‑labels in most settings.
Related stories
arXiv Computation and Language
Oct 27
Adapt or Calibrate? How Local Labels Buy Certified Coverage in Medical VQA
arXiv:2606.15910v3 Announce Type: replace Abstract: An answer from a vision-language model (VLM) to a medical image question can be accepted without review only under a threshold bounding the error o...
arXiv Machine Learning
Sep 30
Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study
arXiv:2609.37848v1 Announce Type: cross Abstract: Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even...
arXiv Computer Vision
Aug 27
Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy
The paper introduces a label‑free method called AURCC for selecting the best foundational model for medical image classification when the target domain lacks labels. AURCC uses a pseudo‑label discrepancy computed by the SUDO framework to score models without fine‑tuning. Experiments on chest X‑ray data across three inter‑hospital shifts show that AURCC closely matches the true model ranking, outperforming simple source‑accuracy baselines especially when source data are limited.
arXiv AI
Oct 12
Medical foundation models converge less under label supervision
The study evaluates how well different medical foundation models agree on image embeddings by measuring mutual k‑nearest‑neighbor overlap across 750,000 images from 14 datasets and 18 public models. Results show that label‑supervised models converge the least, sharing only 6–10% of nearest neighbors, whereas models trained with self‑distillation, masked image modeling, or contrastive pretraining share up to 80%. The authors suggest that developers can select pretraining objectives to control convergence and recommend testing unlabeled images for nearest‑neighbor consistency before replacing a model.
arXiv AI
Jun 17
Vision-language models for chest radiography do not always need the image
arXiv:2606. 17710v1 Announce Type: cross Abstract: Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image.
arXiv AI
Sep 25
Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification and Uncertainty-Aware Evaluation
Med-AR introduces two autoregressive vision‑language models, Med‑AR‑8B and Med‑AR‑2B, pretrained on structured radiology reports, abnormality‑focused text, and region annotations to address long‑tailed chest X‑ray classification. The models outperform existing contrastive, self‑supervised, and supervised encoders—including Med‑CLIP, CheXFound, EVA‑Base, ARK, and BioViL‑T—across PadChest, MIMIC‑CXR, and CheXpert, achieving higher mean AUROC and AUPRC for head, medium, and tail findings and lower excess area under the risk‑coverage curve. Med‑AR also demonstrates improved selective‑prediction performance, with Med‑AR‑8B raising tail‑label mean AUPRC on MIMIC‑CXR from 0.1033 to 0.1441 and Med‑AR‑2B delivering the strongest discrimination on PadChest.
arXiv AI
Oct 27
Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study
The study evaluated ten chest X‑ray AI classification models on 12 datasets from six countries, focusing on image‑level classification and measuring specificity at 95% sensitivity. MAVL and MedKLIP achieved the highest specificity (≈0.37) but still produced many false positives, with performance dropping on institutional cohorts compared to public benchmarks. No single model excelled across all findings, countries, or lesion types, highlighting limited generalisability.
arXiv Computer Vision
6d ago
PRISM: Pathology Reliability In Scarce-label Medicine. A Tile-level Benchmark of Pathology Foundation Models
PRISM is a tile‑level benchmark for pathology foundation models that evaluates reliability across four dimensions: model choice, label fraction, calibration, and out‑of‑distribution transfer. It tests eight models on six datasets at six label fractions (1%–100%) with 864 linear probe runs, plus a five‑hospital Camelyon17 transfer study. The benchmark reveals four key phenomena, including the changing relationship between AUROC and calibration at low labels, collapse of linear probes on harder datasets, the impact of source‑domain labels on target calibration, and limitations of temperature scaling.
arXiv Computation and Language
Aug 21
Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
arXiv:2606. 15910v2 Announce Type: replace Abstract: A vision-language model can answer a question about a chest radiograph or a pathology slide fluently and confidently while barely using the image, relying instead on language priors.
arXiv Machine Learning
Sep 21
Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening
The study evaluates the robustness of medical vision‑language models for tuberculosis screening on chest X‑rays by testing them across multiple datasets, prompts, and evaluation settings. Three specialized models (BioMedCLIP, CheXficient, MedSigLIP) and a general OpenCLIP model were audited on 12,200 images, producing 244,000 model–image–prompt scores. Results show that no model consistently outperforms others across all cohorts and reliability criteria, with prompt changes and control group composition significantly affecting AUROC, and that high training‑set performance does not reliably transfer to external cohorts.
arXiv Machine Learning
Aug 11
Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
arXiv:2608. 07550v1 Announce Type: cross Abstract: Vision-language models return structured chest-radiograph findings through interfaces exposing no confidence score, so a receiving institution cannot read off how far to trust an individual judgment.