Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Methods operating on Vision Transformer (ViT) feature spaces typically rely on Euclidean distance or cosine similarity. This assumes that every direction is equally meaningful, but there is no reason...
arXiv:2606. 02765v1 Announce Type: cross Abstract: Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer language models, yet its role in setting the geometric limits of feature representation remains under-explored.
The paper introduces Kernelized Parameter Sensitivity Atlases (KPSA), a method that uses a block decomposition of the empirical neural tangent kernel to estimate input‑specific parameter sensitivities efficiently. KPSA embeds parameter groups in a query space and applies a lightweight kernel to predict sensitivities for unseen inputs, achieving a high fraction of the perturbation susceptibility of full back‑propagation while dramatically reducing computation and memory usage. Experiments across vision, forecasting, and protein‑modeling tasks show that KPSA identifies sparse, depth‑spanning sensitivity circuits and improves throughput and memory efficiency in activation‑attribution problems.
arXiv:2610.22655v1 Announce Type: cross Abstract: The linear representation hypothesis holds that pretrained vision models encode generative factors as linearly decodable directions in feature space,...
arXiv:2608. 01283v1 Announce Type: new Abstract: All Transformer-based large language models compute attention via the Euclidean inner product, an architectural choice that Dong et al.
arXiv:2609.37717v1 Announce Type: new Abstract: Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state thro...