← Back to all news
arXiv AI By Jan Rathjens, Shirin Reyhanian, David Kappel, Laurenz Wiskott

Understanding Transformer-based Vision Models through Inversion

Read the original on arXiv AI →

The paper revisits feature inversion via inverse networks, proposing a modular variant that is computationally efficient and can produce more semantically coherent image reconstructions. It applies this method to large‑scale transformer‑based vision models—Detection Transformer, Vision Transformer, Swin Transformer, and Data‑Efficient Image Transformer—examining reconstructions across network depth. The analysis highlights gradual representational changes in Detection and Vision Transformers, systematic differences in preserving contextual shape and fine‑grained details, and varying robustness to color perturbations, while also demonstrating the framework’s broader applicability to other transformer architectures.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Jun 18

Investigation of Neural Network Methods for Reconstruction and Classification of Texture Images Under Conditions of Incomplete Information

arXiv:2204. 14224v3 Announce Type: replace-cross Abstract: The automated analysis of heterogeneous natural textures is frequently hindered by physical damage and data loss, presenting a significant challenge to computer vision.

By Galymzhan Abdimanap, Kairat Bostanbekov, Abdelrahman Abdallah, Anel Alimova, Darkhan Kurmangaliyev, Daniyar Nurseitov, Tatyana Dedova, Larissa Balakay, Serik Nurakynov
llmsagentsdiffusioncomputer-visionbenchmarkssafety
More like this →
Hugging Face Trending Papers
Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

llmsefficiencybenchmarkssafety
More like this →
arXiv Machine Learning
Oct 9

Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study

The paper presents a detailed study of feature representations in Vision Transformers (ViTs), revealing that feature collapse at initialization causes redundancy and proposing a reduction scheme to address it. It demonstrates that entropy and minimum eigenvalue metrics reliably predict generalization, and that token‑space features and linear submodule outputs are particularly informative. The authors develop a proxy that improves ranking of ViT architectures by 18‑48% over previous baselines, enabling the selection of models with higher accuracy at comparable or lower computational cost.

By Kim-Cuc Nguyen, Ngai-Man Cheung
llmsragcomputer-vision
More like this →