← Back to all news
arXiv AI By Valeria Ruscio, Keiran Thompson

The Geometry of Learning: Read-Write Asymmetry Across Training

Read the original on arXiv AI →

The paper investigates how pretraining and alignment stages of transformer training affect weight updates in different matrix groups. By examining the residual stream’s shared coordinate system, it finds that alignment updates to query and key matrices align with the principal input subspace, whereas updates to output matrices remain largely random relative to the prediction subspace. This pattern persists across objectives and training stages, indicating that the geometry of updates is determined by the training phase rather than the specific objective.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 22

Reader--Writer Coupling in Transformer MLPs

The paper investigates the dual roles of hidden channels in transformer MLPs, treating each channel as a reader that decides when to respond and a writer that determines what to add back to the residual stream. By examining the joint Jacobian of the MLP’s update, the authors identify a signed pairing coordinate that reveals a small core of strongly coupled reader–writer pairs, often with opposing signs. This opposed core appears early in training and persists across models, influencing how channels contribute to later layers and affecting the model’s output distribution.

By Giyeong Oh, Sunghee Ahn, Youngjae Yu
llms
More like this →
arXiv AI
Oct 16

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.

By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
llms
More like this →
arXiv Computation and Language
Sep 2

Behaviorally Effective LoRA Writes Are Sparse and Structured

The paper investigates which parts of a low‑rank adaptation (LoRA) update actually drive behavior in language models. It shows that behaviorally effective LoRA writes are sparse, structured, and highly concentrated, with only a few module‑wise components carrying most of the signal. Using a learned‑basis continuation recipe, the authors demonstrate that these sparse, structured writes can be recovered with minimal error and that they remain robust across different training continuations and evaluation tasks.

By Haruto Sato, Yuki Tanaka, Ren Nakamura, Aoi Kobayashi, Mei Ito
fine-tuning
More like this →