MVC-Bench is a new benchmark designed to evaluate the calibration of vision‑language models (VLMs) and medical VLMs (Medical‑VLMs) for medical image classification. It tests calibration across robustness to modality, backbone, and domain shift; effectiveness of calibration strategies and prompt‑tuning methods; and stability under prompt‑template and random‑seed variations. The benchmark includes eight backbones, three medical modalities (fundus imaging, histopathology, chest X‑ray), and compares post‑hoc, train‑time, and zero‑shot calibration approaches, reporting accuracy, Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Adaptive Calibration Error (ACE) over 1,638 experiments, while also proposing a Multi‑Class Margin (MCM) regularization technique that improves ECE in most settings.
By Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan, Muhammad Akhtar Munir Sujair Ibrahim, Mohamed Rafeek Mareer Ahamed, Yutong Xie, Imran Razzak, Muhammad Haris Khan
The paper presents DualTTA, a model‑agnostic framework that improves the calibration of black‑box radiology AI systems by applying clinically grounded test‑time augmentations (geometric and physics‑inspired 3D CT perturbations) and learning probability‑level aggregation strategies. Without accessing model internals or training data, DualTTA achieved the best overall calibration across pulmonary embolism and intracranial hemorrhage detection tasks, reducing Expected Calibration Error by 54% and 43% respectively. It also outperformed traditional uncertainty estimation methods that require internal model access, such as Temperature Scaling, MC Dropout, and Deep Ensembles.
By Nathan Le, Magdalini Paschali, Arogya Koirala, Andrew Johnston, Zhongnan Fang, David B. Larson, Akshay S. Chaudhari, Camila Gonzalez
MVC-Bench is a calibration-focused benchmark for medical vision‑language models, evaluating how well these models express confidence across different modalities, backbones, and domain shifts. It tests robustness to modality, backbone, and domain changes, the effectiveness of calibration and prompt‑tuning strategies, and stability under prompt‑template and random‑seed variations. The benchmark includes 1638 experiments, reporting accuracy and Expected Calibration Error (ECE) along with other calibration metrics, and introduces a simple train‑time calibration method, Multi‑Class Margin (MCM) regularization, that achieves the lowest ECE in most settings.
Automated radiology report generation is advancing rapidly in response to the shortage of radiologists, yet unlike a perception model, existing generation models offer no control over the sensitivity-specificity trade-off of their diagnostic content. Such control is essential because clinical scenarios diverge: emergency triage prioritizes sensitivity to reduce missed findings, whereas confirmatory interpretation emphasizes specificity to limit unnecessary interventions.
arXiv:2609.26468v1 Announce Type: new
Abstract: A key factor in deciding whether to trust an automatic prediction is its confidence score, which should be calibrated to match the actual probability o...
By Sophie Henning, Georg Hofmann, Alexander Schulte, Alexander Fraser, Annemarie Friedrich
arXiv:2608. 10505v1 Announce Type: new Abstract: Automated radiology report generation is advancing rapidly in response to the shortage of radiologists, yet unlike a perception model, existing generation models offer no control over the sensitivity-specificity trade-off of their diagnostic content.
By Ying Jin, Noel C. F. Codella, John Corring, Mu Wei, Dinei Florencio, Eric Horvitz
arXiv:2605.30984v2 Announce Type: replace-cross
Abstract: Modern 3D medical vision-language models (VLMs) can generate fluent radiology-style text while exhibit critically low pathology detection and...
By Tom Maye-Lasserre, Yitong Li, Bailiang Jian, Morteza Ghahremani, Benedikt Wiestler, Christian Wachinger
arXiv:2609.01470v1 Announce Type: new
Abstract: As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language...
By Charles Corbi\`ere, L\'eo Machado, Aubin Charley, Baptiste Callard, Pierre Manceron, Corentin Dancette
The paper presents an optimal‑transport based generative model that learns the distributional differences between healthy and diseased patients, producing per‑patient counterfactuals and label‑free attribution heatmaps. On tabular breast cancer data the model achieves high malignancy scoring (AUROC ≈ 0.91) and its attributions correlate moderately with a supervised classifier, yet it does not surpass logistic regression. In chest X‑ray experiments the transport heatmaps capture population‑level signals but fail to localize real lesions, revealing a synthetic‑to‑real gap that challenges the reliability of label‑free explanations.
By Lalit Kumar
CasCVS‑Net is a staged multi‑task cascade that jointly performs object detection, semantic segmentation, and Critical View of Safety (CVS) assessment for laparoscopic cholecystectomy. The model couples tasks through predicted anatomy—boxes guide segmentation and masks provide region‑level features for CVS classification—allowing CVS assessment to rely solely on model predictions. Trained on the Endoscapes dataset, CasCVS‑Net outperforms state‑of‑the‑art methods, achieving higher mAP and mIoU scores across detection, segmentation, and CVS tasks, especially for rare hepatocystic structures.
By Bock-Zien Toh, Yuanchuan Ren, Tay Aw Yu, Ng Khee Ong, Zhehua Mao, Sophia Bano
arXiv:2606. 15910v2 Announce Type: replace Abstract: A vision-language model can answer a question about a chest radiograph or a pathology slide fluently and confidently while barely using the image, relying instead on language priors.
By Reza Khanmohammadi, Kundan Thind, Mohammad M. Ghassemi
arXiv:2607. 13423v1 Announce Type: new Abstract: Temperature scaling is the dominant post-hoc calibration method in modern deep learning.
By Wisdom Dogah