arXiv AI

ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

arXiv Machine Learning
Oct 23

Measuring Pretraining Contamination in Public Medical Vision-Language Benchmarks

The paper investigates whether public medical vision‑language benchmarks contain data that may have leaked into the training sets of open VLMs. Using four contamination detectors, the authors find that a significant portion of images in SLAKE‑En have close neighbours in PMC‑OA‑beta, indicating source overlap rather than exact memorization. On the text side, only the exchangeability detector consistently flags potential contamination, while the other methods produce false positives and are unreliable without external baselines.

By Bruce Changlong Xu, Lan Wu, Alexander Ryu
arXiv Computation and Language
Oct 16

Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis

The study identifies a specific attention head (L8H10) in Vision Transformers that reads annotation text on chest X‑ray images instead of lung pathology, affecting 62% of COVID‑19 images. Causal inpainting experiments confirm that the head’s behavior is driven by annotation content, and removing the head increases lung attention by 1.52×. This spurious annotation‑attending behavior is consistent across multiple ViT architectures and datasets, persists after full fine‑tuning, and is invisible to standard output‑level evaluation, highlighting the need for mechanistic auditing.

By Joshua Muhumuza, Cornelius Kalenzi