arXiv Computation and Language

Through the Looking Glass: Directly Reading and Writing Transformers

The paper investigates how many transformer components influence a token prediction by measuring the absolute contribution of each unit and channel to the logit. It finds that thousands of components contribute to a single prediction, yet a small subset—often just dozens—carries the majority of the predictive mass. Across models ranging from 124 M to 7 B parameters, the proportion of the model involved in a prediction remains around one to three percent, independent of size, and the study demonstrates that specific components can be directly read and written to modify model behavior without additional training.

arXiv Machine Learning
Jul 13

Training, Reading, and Editing Legible Transformers

arXiv:2607. 08946v1 Announce Type: new Abstract: A transformer can be built from operators that are legible by construction -- bounded, named units that read as fuzzy set operations rather than dense activations -- but legibility must be pressed for during training, and the pressure has a failure mode.

By Mark Oskin
arXiv Machine Learning
Oct 22

Reader--Writer Coupling in Transformer MLPs

The paper investigates the dual roles of hidden channels in transformer MLPs, treating each channel as a reader that decides when to respond and a writer that determines what to add back to the residual stream. By examining the joint Jacobian of the MLP’s update, the authors identify a signed pairing coordinate that reveals a small core of strongly coupled reader–writer pairs, often with opposing signs. This opposed core appears early in training and persists across models, influencing how channels contribute to later layers and affecting the model’s output distribution.

By Giyeong Oh, Sunghee Ahn, Youngjae Yu
arXiv Computation and Language
Aug 25

The Communication Map of a Transformer

The paper introduces the Communication Map, a method that charts every potential communication channel in a transformer model using only its weights. It generalizes previous coupling metrics into a single coefficient covering all 18 connection classes, revealing that 70‑89% of head pairs are non‑randomly oriented and identifying strong or avoiding couplings. The authors demonstrate the map’s utility by recovering known induction circuits and uncovering a two‑dimensional stream subspace whose removal eliminates induction capabilities across several models.

By Richard Zhe Wang
arXiv AI
Sep 25

Every Component Is a Lookup: One Linear Graph for Interaction, Composition and Attribution

The paper proposes that two architectural assumptions—(1) attention and MLPs share a key‑value form <phi(S)>U, and (2) components read from an additive residual stream—are sufficient to answer three interpretability questions: component interaction, information routing, and token attribution. By treating these selections as a computational graph, the authors develop Unpack, a backward attribution method that validates interaction scores, recovered routes, and token attribution against established tests across models ranging from 160M to 6.9B parameters. The study also shows that contribution and causal effect can differ, with a recognizable signature in how components change when a task is removed.

By Po-Kai Chen, Aske Plaat, Niki van Stein
arXiv AI
Oct 27

Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention

The paper investigates how treating each entity in a graph or language model as having its own coordinate frame—viewed as a symmetry—constrains the form of attention mechanisms. It shows that with orthogonal transports, invariant attention scores depend only on norms and transported inner products, while equivariant messages lie in the span of the entity vectors and their transported counterparts. The authors demonstrate that standard attention corresponds to a flat shared frame, and that incorporating per‑entity frames allows relations to be represented as operators that compose via parallel transport, yielding empirical gains on synthetic relational tasks and natural language benchmarks.

By Aakash Sarkar, Ananya Mukherjee, Motoshige Sato
arXiv Machine Learning
Oct 16

Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches

The paper investigates whether a frozen transformer can use another model’s KV cache as a substitute for its own, finding that this approach fails because the reader only accepts caches within a narrow tolerable set. Experimental results show that none of eight substitution methods improve performance over a baseline, while a different strategy—adding the sender’s cache as extra context and only adapting queries—does outperform the baseline. The authors conclude that pursuing faithful cache substitution is misguided and that augmenting the reader’s own cache is the more promising direction.

By Adam Deng, Shicheng Liu, Yada Zhu