arXiv Computer Vision

Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification

HierViT is a hierarchical Vision Transformer designed for medical image classification that balances strong predictive performance with interpretability. The model first predicts clinically relevant visual appearance attributes and then uses a hierarchical reasoning process to infer the final diagnosis. Attribute‑specific prototypes and attention maps are visualized to provide complementary views of the model’s reasoning, linking representative image examples and spatial attention to semantically defined visual features.

arXiv AI

5d ago

A Lightweight Attention-Based CNN with Saliency-Map Interpretability for Skin Cancer Detection

The paper introduces a lightweight attention-based CNN (CNN-Attention) for binary skin cancer classification that uses channel attention to highlight informative dermoscopic regions while keeping computational demands low. Trained on a dataset of 2,637 images, the model achieved 87 % accuracy, 0.9475 ROC AUC, and 0.9430 PR AUC, matching MobileNet’s performance with roughly half the parameters and storage. Saliency maps are employed to provide explainability by visualizing the image areas that influence predictions.

By Sanjana Islam Kasfia, Md. Ismiel Hossen Abir, AKM Bahalul Haque, Abir Mahmud Shahariar

safety

arXiv AI

Oct 7

MedCORE: Criteria-Grounded Clinical Reasoning for Interpretable Medical Image Diagnosis

arXiv:2610.08528v1 Announce Type: cross Abstract: Clinical diagnosis is inherently a structured reasoning process, yet existing deep learning models often bypass this structure by mapping image featu...

By Asim Khan, Samee Ullah Khan, Dwarikanath Mahapatra

llms multimodal

arXiv AI

Sep 30

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

arXiv:2609.36557v1 Announce Type: cross Abstract: Medical Vision-Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable reaso...

By Janet Wang, Yunbei Zhang, Xiao Wang, Jihun Hamm

llms rag fine-tuning multimodal

arXiv Machine Learning

Jul 10

A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

arXiv:2512. 21414v2 Announce Type: replace-cross Abstract: Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools.

By Christina Liu, Alan Q. Wang, Joy Hsu, Jiajun Wu, Ehsan Adeli

llms multimodal benchmarks

arXiv AI

Jun 9

MedVision: Benchmarking Quantitative Medical Image Analysis

arXiv:2511. 18676v2 Announce Type: replace-cross Abstract: Current vision-language models (VLMs) in medicine are primarily designed for categorical question answering (e.

By Yongcheng Yao, Yongshuo Zong, Raman Dutt, Yongxin Yang, Sotirios A Tsaftaris, Timothy Hospedales

llms nlp fine-tuning multimodal benchmarks

arXiv AI

6d ago

SAP-ViT: A Self-Attributing Patch Vision Transformer for Medical Image Classification

SAP-ViT is a Vision Transformer that replaces the traditional global classification token with direct aggregation of last‑layer patch embeddings, enabling class predictions to be decomposed into patch‑level contributions during the forward pass. The model incorporates sparsity regularization to produce concise, readable evidence and introduces an input‑space back‑mapping mechanism to identify which input patches influence the output. Experiments on diverse medical imaging datasets show that SAP‑ViT retains strong classification performance while delivering consistent, interpretable patch‑level explanations, with ablation studies confirming the benefits of both sparsity and back‑mapping.

By Wei Zhang, Wolfgang Nejdl, Ming Tang

llms rag computer-vision efficiency benchmarks safety

arXiv AI

Sep 10

DL$^3$M: A Vision-to-Language Framework for Expert-Level Medical Reasoning through Deep Learning and Large Language Models

arXiv:2512.13742v3 Announce Type: replace-cross Abstract: Medical image classifiers detect gastrointestinal diseases well, but they do not explain their decisions. Large language models can generate...

By Md. Najib Hasan (Wichita State University, USA), Imran Ahmad (Wichita State University, USA), Sourav Basak Shuvo (Khulna University of Engineering and Technology, Bangladesh), Md. Mahadi Hasan Ankon (Khulna University of Engineering and Technology, Bangladesh), Nazmul Siddique (Ulster University, UK), Hui Wang (Queen's University Belfast, UK)

llms computer-vision benchmarks

arXiv Computer Vision

Sep 10

RAU: Reference-based Anatomical Understanding with Vision Language Models

arXiv:2509.22404v2 Announce Type: replace Abstract: Anatomical understanding, which is the ability to identify, localize, or segment anatomical structures, is critical in medical image analysis; howe...

By Yiwei Li, Yikang Liu, Jiaqi Guo, Lin Zhao, Zheyuan Zhang, Xiao Chen, Boris Mailhe, Ankush Mukherjee, Terrence Chen, Shanhui Sun

llms computer-vision nlp fine-tuning multimodal

arXiv Machine Learning

Sep 23

Faithful Faithfulness Evaluations: Challenges & Pitfalls Learned from a Breast MRI Case Study

arXiv:2609.25978v1 Announce Type: cross Abstract: Saliency maps are widely used to explain deep learning predictions in medical imaging, yet visually plausible explanations do not necessarily reflect...

By Peachapong Poolpol, Henrik H. J. Detjen, Eike Petersen

llms computer-vision benchmarks

arXiv Computer Vision

Sep 1

Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification

arXiv:2607.10851v2 Announce Type: replace Abstract: Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classif...

By Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali, Miaomiao Zhang

computer-vision safety

arXiv Computer Vision

Sep 10

Synergistic Vision-Language Reinforcement Enables Scalable On-Demand Analysis across Diverse Clinical Tasks

arXiv:2505.03380v2 Announce Type: replace Abstract: Accurate delineation of tumors and surrounding organs-at-risk is essential for radiotherapy, surgery and treatment response assessment, yet remains...

By Haonan Wang, Jiaji Mao, Lehan Wang, Qixiang Zhang, Marawan Elbatel, Yi Qin, Huijun Hu, Baoxun Li, Wenhui Deng, Weifeng Qin, Hongrui Li, Jialin Liang, Jun Shen, Xiaomeng Li

llms computer-vision multimodal

arXiv AI

Jun 11

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

arXiv:2606. 12169v1 Announce Type: cross Abstract: High-stakes clinical use of large vision-language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers.

By Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci, Abeer Badawi, Adibvafa Fallahpour, Arash Afkanpour, Leonid Sigal, Ali Etemad, Elham Dolatabadi

llms fine-tuning multimodal benchmarks safety

Related stories