arXiv Machine Learning July 14, 2026 By Matthias M. M. Buehlmaier

LayerNorm as Implicit Gain Control in Looped Transformers

Read the original on arXiv Machine Learning →

arXiv:2607. 10681v1 Announce Type: new Abstract: In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 16

DeepLoop: Depth Scaling for Looped Transformers

arXiv:2607. 13491v1 Announce Type: cross Abstract: Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters.

By Shuzhen Li, Yifan Zhang, Jiacheng Guo, Quanquan Gu, Mengdi Wang

llms safety

arXiv Machine Learning

Jun 18

On the Residual Scaling of Looped Transformers: Stability and Transferability

arXiv:2606. 18524v1 Announce Type: new Abstract: Looped (weight-tied) Transformers apply a shared residual block $N$ times ($h \leftarrow h + \varepsilon\,f(h)$, same $f$ at each step), increasing effective depth without adding parameters.

By Shaowen Wang, Bingrui Li, Ge Zhang, Wenhao Huang, Shen Yan, Jian Li

llms

arXiv Machine Learning

Aug 6

Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision

arXiv:2608. 04879v1 Announce Type: new Abstract: Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized.

By Grzegorz Gruszczynski, Pawel Olszowiec, Michal Byra, Grzegorz Stefanski, Alberto Presta

llms safety

Hugging Face Trending Papers

Jul 23

Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well.

llms

arXiv AI

Jul 24

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

arXiv:2607. 20594v1 Announce Type: cross Abstract: When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm?

By Tong Zhang, Junhao Hu, Yun Peng, Tao Xie

llms benchmarks

arXiv AI

Jul 24

Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

arXiv:2607. 21000v1 Announce Type: new Abstract: Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones.

By Hyuk Lim, Seunghyun Yoon

llms

Related stories

DeepLoop: Depth Scaling for Looped Transformers

On the Residual Scaling of Looped Transformers: Stability and Transferability

Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision

Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory