The paper investigates the dual roles of hidden channels in transformer MLPs, treating each channel as a reader that decides when to respond and a writer that determines what to add back to the residual stream. By examining the joint Jacobian of the MLP’s update, the authors identify a signed pairing coordinate that reveals a small core of strongly coupled reader–writer pairs, often with opposing signs. This opposed core appears early in training and persists across models, influencing how channels contribute to later layers and affecting the model’s output distribution.
By Giyeong Oh, Sunghee Ahn, Youngjae Yu
arXiv:2608. 12447v1 Announce Type: new Abstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream.
By Nelson Guda
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.
By Devender Singh
arXiv:2606. 31963v1 Announce Type: new Abstract: Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, and merge alignments.
By John Sweeney
The paper investigates which parts of a low‑rank adaptation (LoRA) update actually drive behavior in language models. It shows that behaviorally effective LoRA writes are sparse, structured, and highly concentrated, with only a few module‑wise components carrying most of the signal. Using a learned‑basis continuation recipe, the authors demonstrate that these sparse, structured writes can be recovered with minimal error and that they remain robust across different training continuations and evaluation tasks.
By Haruto Sato, Yuki Tanaka, Ren Nakamura, Aoi Kobayashi, Mei Ito
The paper introduces a stability theory for Joint-Embedding Predictive Architectures (JEPAs), showing that training dynamics involve a driving force and a decay effect that determine representation collapse. By linearising the gradient flow, the authors derive a per‑mode stability ratio that separates data‑side and predictor‑side contributions, predicting a phase boundary confirmed across 800 configurations. Using this insight, they propose ResidualPred, a transformer predictor that biases attention toward the identity at initialization, improving representation rank and downstream accuracy on tabular and image benchmarks.
By Jos\'e Lucas De Melo Costa, Seong Woo Ahn, Fabrice Popineau, Arpad Rimmel, Bich-Li\^en Doan
arXiv:2608. 10251v1 Announce Type: cross Abstract: A transformer's answer lives on one axis: the direction its unembedding reads.
By Mark Oskin
arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.
By Irina Piontkovskaia, Sergey Nikolenko
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes.
whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
By Hao Wu
Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained
write actually carry behavior. We study that question directly and show that behaviorally effectiv...
arXiv:2610.16268v2 Announce Type: replace
Abstract: Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself...
By Yuhe Hu