arXiv Machine Learning
How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
4d ago
Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
arXiv:2610.26030v2 Announce Type: replace Abstract: Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are...
arXiv AI
Oct 15
Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models
arXiv:2610.15218v1 Announce Type: new Abstract: A recurrent-depth language model reuses one block of weights several times in a row. The number of repetitions, which we call loops, is chosen when the...
arXiv Machine Learning
Oct 15
Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
arXiv:2610.19768v1 Announce Type: new Abstract: Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known...
arXiv Machine Learning
6d ago
Flat Loss, Evolving Features: Grokking in Modular Addition
The paper demonstrates that neural networks can maintain perfect training accuracy while their training cross‑entropy remains nearly flat at a positive level, a phenomenon observed on modular addition tasks. Using ReLU networks with supported‑Polar hidden updates, normalized output‑layer updates, and weight decay, the authors show that hidden features evolve and held‑out accuracy can rise from zero to above 90% even as cross‑entropy stays flat. Theoretical results for one‑hidden‑layer networks and empirical evidence from deeper networks and transformers confirm that correct classification need not coincide with minimal cross‑entropy, challenging conventional assumptions about loss plateaus.
arXiv Machine Learning
Sep 18
Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not
The paper introduces Stiefel Attention, which constrains the query and key projection matrices of transformers to the Stiefel manifold and optimizes them with a Riemannian Adam variant. It demonstrates that this approach yields steepest‑descent updates, is well‑conditioned, and preserves learned attention geometry during weight decay. Empirical results show significant accuracy gains on modular arithmetic grokking and CIFAR‑10 patches, with the improvement attributed to a step‑scale‑free update rule rather than equivariance or projector changes.
arXiv Machine Learning
Oct 2
Backward-State Policy Is Part of the Learning Algorithm
The paper argues that the policy governing how tensors are rounded and reused during the backward pass—termed the backward‑state policy—is an integral part of the learning algorithm, not merely a memory or precision detail. Experiments with 390 M‑parameter models show that whether the backward pass reuses a forward’s rounded output or generates a new rounding can decisively affect training success, even when copy accuracy is high. The authors propose a method to determine, for each operator, which value should be read or substituted to preserve the correct gradient, and validate these predictions on PyTorch and Transformer Engine.