arXiv Machine Learning
Jun 3
Attribution via Distributional Paths for Information Revelation
arXiv:2606. 03885v1 Announce Type: new Abstract: Feature attribution methods explain predictions by assigning importance scores to input features.
Interior interpretability with attention rollout: contraction and propagation profiles in Transformers
arXiv:2607. 22367v1 Announce Type: new Abstract: Feature-attribution methods assign scores relating input variables to a model's output, but do not by themselves characterize how explicitly defined interaction operators compose across its intermediate layers.
Related stories
arXiv AI
Jun 2
A Monosemantic Attribution Framework for Stable Interpretability in Clinical Neuroscience Transformer-Based Language Models
arXiv:2601. 17952v2 Announce Type: replace-cross Abstract: Interpretability remains a key challenge for deploying language models (LM) in clinical settings such as progression diagnosis of Alzheimer disease, where early and trustworthy predictions are essential.
arXiv Machine Learning
6d ago
Geometric and Behavioral Stratification in Transformer Residual Streams
arXiv:2608. 12447v1 Announce Type: new Abstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream.
arXiv Machine Learning
Jun 30
Generalization Analysis of Transformers in Distribution Regression
arXiv:2606. 29256v1 Announce Type: cross Abstract: In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning.
arXiv Machine Learning
Aug 11
A Mechanistic Diagnostic of Rank Collapse in Post-Norm Decoder Transformers
arXiv:2608. 09417v1 Announce Type: new Abstract: Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes.
arXiv AI
Jul 23
Geometric Attention: A Regime-Explicit Operator Semantics for Transformer Attention
arXiv:2601. 11618v2 Announce Type: replace-cross Abstract: Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied).
arXiv AI
Jun 30
Feature-level Interaction Explanations in Multimodal Transformers
arXiv:2603. 13326v2 Announce Type: replace-cross Abstract: Multimodal Transformers often produce predictions without clarifying how different modalities jointly support a decision.
arXiv Machine Learning
Jun 8
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
arXiv:2604. 10098v2 Announce Type: replace Abstract: As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains.
arXiv AI
Jun 2
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
arXiv:2605. 27458v2 Announce Type: replace-cross Abstract: Transformer has significantly propelled the development of artificial intelligence, and certainly the development of agents as well.
arXiv Machine Learning
Jul 22
Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
arXiv AI
Jun 16
DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing
arXiv:2606. 15796v1 Announce Type: cross Abstract: Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits.
arXiv AI
2d ago
Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data
arXiv:2608. 14712v1 Announce Type: cross Abstract: Each row of a transformer's attention matrix is a probability distribution over tokens, and in trained models most of that probability lands on a single \emph{sink} token, usually the first.