arXiv Computer Vision
SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Computer Vision
Aug 28
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
MVC-Bench is a new benchmark designed to evaluate the calibration of vision‑language models (VLMs) and medical VLMs (Medical‑VLMs) for medical image classification. It tests calibration across robustness to modality, backbone, and domain shift; effectiveness of calibration strategies and prompt‑tuning methods; and stability under prompt‑template and random‑seed variations. The benchmark includes eight backbones, three medical modalities (fundus imaging, histopathology, chest X‑ray), and compares post‑hoc, train‑time, and zero‑shot calibration approaches, reporting accuracy, Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Adaptive Calibration Error (ACE) over 1,638 experiments, while also proposing a Multi‑Class Margin (MCM) regularization technique that improves ECE in most settings.
arXiv Machine Learning
Sep 25
Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentation
The paper presents DualTTA, a model‑agnostic framework that improves the calibration of black‑box radiology AI systems by applying clinically grounded test‑time augmentations (geometric and physics‑inspired 3D CT perturbations) and learning probability‑level aggregation strategies. Without accessing model internals or training data, DualTTA achieved the best overall calibration across pulmonary embolism and intracranial hemorrhage detection tasks, reducing Expected Calibration Error by 54% and 43% respectively. It also outperformed traditional uncertainty estimation methods that require internal model access, such as Temperature Scaling, MC Dropout, and Deep Ensembles.
Hugging Face Trending Papers
Aug 27
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
MVC-Bench is a calibration-focused benchmark for medical vision‑language models, evaluating how well these models express confidence across different modalities, backbones, and domain shifts. It tests robustness to modality, backbone, and domain changes, the effectiveness of calibration and prompt‑tuning strategies, and stability under prompt‑template and random‑seed variations. The benchmark includes 1638 experiments, reporting accuracy and Expected Calibration Error (ECE) along with other calibration metrics, and introduces a simple train‑time calibration method, Multi‑Class Margin (MCM) regularization, that achieves the lowest ECE in most settings.
Hugging Face Trending Papers
Aug 11
RadFusion: Towards Threshold-Controllable Radiology Report Generation
Automated radiology report generation is advancing rapidly in response to the shortage of radiologists, yet unlike a perception model, existing generation models offer no control over the sensitivity-specificity trade-off of their diagnostic content. Such control is essential because clinical scenarios diverge: emergency triage prioritizes sensitivity to reduce missed findings, whereas confirmatory interpretation emphasizes specificity to limit unnecessary interventions.
arXiv Computation and Language
Sep 23
How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification
arXiv:2609.26468v1 Announce Type: new Abstract: A key factor in deciding whether to trust an automatic prediction is its confidence score, which should be calibrated to match the actual probability o...
arXiv AI
Aug 12
RadFusion: Towards Threshold-Controllable Radiology Report Generation
arXiv:2608. 10505v1 Announce Type: new Abstract: Automated radiology report generation is advancing rapidly in response to the shortage of radiologists, yet unlike a perception model, existing generation models offer no control over the sensitivity-specificity trade-off of their diagnostic content.