← Back to all news
arXiv Computer Vision By Jinyeong Kim, Junhyeok Kim, Yumin Shim, Joohyeok Kim, Sunyoung Jung, Seong Jae Hwang

Interpreting vision transformers via residual replacement model

Read the original on arXiv Computer Vision →

The paper investigates how vision transformers (ViTs) represent and process visual information by conducting a systematic analysis of 6.6K features across all layers using sparse autoencoders. It introduces the residual replacement model, which substitutes ViT computations with interpretable features in the residual stream, revealing a progression from low-level patterns to high-level semantics and specialized encoding of curves and spatial positions. This model provides a scalable, faithful, and parsimonious circuit that simplifies ViT computations, enabling intuitive understanding of ViT mechanisms and demonstrating its usefulness for debiasing spurious correlations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Machine Learning
Oct 9

Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study

The paper presents a detailed study of feature representations in Vision Transformers (ViTs), revealing that feature collapse at initialization causes redundancy and proposing a reduction scheme to address it. It demonstrates that entropy and minimum eigenvalue metrics reliably predict generalization, and that token‑space features and linear submodule outputs are particularly informative. The authors develop a proxy that improves ranking of ViT architectures by 18‑48% over previous baselines, enabling the selection of models with higher accuracy at comparable or lower computational cost.

By Kim-Cuc Nguyen, Ngai-Man Cheung
llmsragcomputer-vision
More like this →
arXiv AI
Oct 12

Understanding Transformer-based Vision Models through Inversion

The paper revisits feature inversion via inverse networks, proposing a modular variant that is computationally efficient and can produce more semantically coherent image reconstructions. It applies this method to large‑scale transformer‑based vision models—Detection Transformer, Vision Transformer, Swin Transformer, and Data‑Efficient Image Transformer—examining reconstructions across network depth. The analysis highlights gradual representational changes in Detection and Vision Transformers, systematic differences in preserving contextual shape and fine‑grained details, and varying robustness to color perturbations, while also demonstrating the framework’s broader applicability to other transformer architectures.

By Jan Rathjens, Shirin Reyhanian, David Kappel, Laurenz Wiskott
llmscomputer-visionsafety
More like this →