arXiv Computation and Language October 16, 2026 By Joshua Muhumuza, Cornelius Kalenzi

Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis

Read the original on arXiv Computation and Language →

The study identifies a specific attention head (L8H10) in Vision Transformers that reads annotation text on chest X‑ray images instead of lung pathology, affecting 62% of COVID‑19 images. Causal inpainting experiments confirm that the head’s behavior is driven by annotation content, and removing the head increases lung attention by 1.52×. This spurious annotation‑attending behavior is consistent across multiple ViT architectures and datasets, persists after full fine‑tuning, and is invisible to standard output‑level evaluation, highlighting the need for mechanistic auditing.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Oct 1

CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models

arXiv:2609.38810v1 Announce Type: cross Abstract: As vision language models are increasingly deployed in clinical diagnosis, understanding how they internally resolve competing visual and textual sig...

By Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao, Yihang Chen, Yijun Wang, Jianxin Lin

llms multimodal benchmarks safety

arXiv AI

Jun 17

Vision-language models for chest radiography do not always need the image

arXiv:2606. 17710v1 Announce Type: cross Abstract: Medical vision-language models report strong chest radiograph accuracy, and this is increasingly read as evidence that they use the image.

By Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh

llms multimodal benchmarks

arXiv AI

Jun 10

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

arXiv:2606. 10066v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years, yet reported accuracy assumes these examples were absent from pretraining.

By Bruce Changlong Xu, Lan Wu, Alexander Ryu

llms multimodal benchmarks

arXiv Computer Vision

Aug 27

Targeting the Attention Heads Behind Object Hallucination in LLaVA

The paper investigates why vision‑language models like LLaVA‑1.5‑7B hallucinate objects in captions and proposes a targeted fix. By ranking attention heads whose image attention drops around hallucinated words, the authors identify 32 key heads and apply a head‑sliced LoRA adapter plus an inference‑time grounding controller. On COCO images, this combined method reduces hallucinated captions from 37% to 23% and hallucinated object mentions from 15.6% to 9.6%, while also lowering object recall.

By Armaan Sandhu, Abhilasha Senapati, Hima Kammachi

llms fine-tuning multimodal safety

Hugging Face Trending Papers

Sep 30

CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models

As vision language models are increasingly deployed in clinical diagnosis, understanding how they internally resolve competing visual and textual signals becomes a safety imperative. Existing mechanis...

llms multimodal benchmarks safety

arXiv AI

Sep 15

ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

arXiv:2609.15635v1 Announce Type: cross Abstract: A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image. ModaLens, a pai...

By Sebasti\'an Andr\'es Cajas Ord\'o\~nez, Maximin Lange, Quang Bui, Anqi Peter Li, Felipe Ocampo Osorio, Rafi Al Attrach, Kushul Reddy Palakala, Sahil Kapadia, Zakaria Laouabdia Sellami, Xinyue Zhang, Ashley Zhang, Leo Anthony Celi

llms multimodal

One email a morning, machine-written

Related stories

CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models

Vision-language models for chest radiography do not always need the image

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

Targeting the Attention Heads Behind Object Hallucination in LLaVA

CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models

ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs