Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation
Read the original on arXiv Machine Learning →The paper presents a quantitative study of how well the attention maps of CNNs, Vision Transformers, and Vision Mamba align with human visual perception, using eye‑tracking data as ground truth. A standardized normalization pipeline maps attribution maps and gaze heatmaps to spatial probability distributions, and alignment is measured with Jensen‑Shannon similarity, Chi‑square similarity, and Pearson correlation across fine‑grained classification, medical imaging, and emotion recognition tasks. Results show that CNNs exhibit more global, diffuse saliency that generally matches human perception, ViTs focus on fine‑grained, patch‑level features, and Vision Mamba performs well in medical tasks but diverges in natural images, with architectural and training choices affecting interpretability.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.