arXiv AI
Aug 21
Geometric Evolution Maps: Extracting Stable Concept Probes from Transformer Residual Streams
arXiv:2605. 25848v2 Announce Type: replace-cross Abstract: A concept probe is only as reliable as the layer it is taken from.
The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth
arXiv:2605. 24856v2 Announce Type: replace-cross Abstract: Concept formation in transformer language models is a depth-extended process, not a single-layer event: a concept becomes separable across one or more contiguous regions of the residual stream - its Concept Allocation Zone (CAZ).
Related stories
arXiv Machine Learning
Aug 12
Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So
arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.
arXiv AI
Jul 14
Laguerre Geometry for Interpreting Large Language Models
arXiv:2607. 10578v1 Announce Type: new Abstract: Existing hypotheses represent a concept in an LLM as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge.
arXiv AI
Jun 12
Bag of Dims: Training-Free Mechanistic Interpretability via Dimension-Level Sign Patterns
arXiv:2606. 12629v1 Announce Type: cross Abstract: We show that the standard basis of transformer hidden states already provides a training-free, architecture-general feature basis.
arXiv Machine Learning
Sep 30
The Geometry of Inference in Transformer Residual Streams
arXiv:2609.37824v1 Announce Type: cross Abstract: Transformer language models build predictions through successive residual updates, but how their representations become specific to an eventual outco...
arXiv Machine Learning
Sep 21
Exemplar Partitioning for Mechanistic Interpretability
The paper introduces Exemplar Partitioning (EP), an unsupervised technique that constructs interpretable feature dictionaries from large language model activations by clustering streamed activations into Voronoi regions defined by exemplars and their averages. EP allows comparison of dictionaries across layers, checkpoints, and architectures, and demonstrates utility in interpreting model behavior, tracking training dynamics, detecting hidden concepts, and enabling targeted interventions. Experiments on Gemma‑2‑2B and Llama‑3.1‑8B show EP can reveal how instruction tuning reorganizes harmful prompt activations, facilitate interventions that alter model responses, and achieve high concept‑detection performance while requiring far fewer construction tokens than comparable methods.
arXiv AI
Jul 22
An Analysis of Residual-Stream Geometry Across Transformer Depth
arXiv:2607. 18348v1 Announce Type: cross Abstract: We propose a transition-centred geometric analysis of transformer residual streams.
arXiv AI
Jun 8
Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers
arXiv:2606. 06664v1 Announce Type: cross Abstract: Despite high accuracy, Vision Transformer (ViT) predictions can be driven by spurious cues, raising the need to understand their inner workings before safe deployment.
arXiv AI
Jul 9
The Signs Were Always There: Training-Free Concept Detection and Steering in Raw Transformer Dimensions
arXiv:2606. 12629v3 Announce Type: replace-cross Abstract: The standard basis of transformer hidden states is a training-free, architecture-general feature basis for detecting concepts and, in language models, steering them; with no learned dictionary.
arXiv Machine Learning
Jun 29
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
arXiv AI
Jun 9
Phantom transitions in language model fine-tuning
arXiv:2606. 07559v1 Announce Type: cross Abstract: Fine-tuning a language model on contexts whose correct completion has a near-synonym competitor often fails silently.
arXiv AI
Sep 2
Are You Thinking What I am Thinking? : Examining Conceptual Separation in Neural Architectures
The paper investigates whether neural networks exhibit conceptual separation, meaning that examples of the same concept cluster together and related concepts are closer in representation space. Using geometric and distributional analyses, the authors find that Convolutional Neural Networks (CNNs) produce coherent, semantically ordered representations for familiar ImageNet concepts, but this coherence weakens for unseen concepts and under domain shift. Large Language Models (LLMs) keep distinct domains well separated, bring related subdomains closer, yet lose distinction between ambiguous topics at both mean and covariance levels.