arXiv AI October 9, 2026 By Zirui Peng, Yizhou Liu, Ziming Liu, Jeff Gore

Emergent Inverse-Depth Scaling From Nonlinearity In Attention

The paper investigates how nonlinearity in attention mechanisms affects scaling laws in large language models. It finds that nonlinear attention leads to an inverse-depth decay of loss across various data spectra, enabling models to learn both strong and weak spectral directions simultaneously. This behavior contrasts with linear-attention models, which rely on a power-law data spectrum and learn directions sequentially.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Jun 2

Inverse Depth Scaling From Most Layers Being Similar

arXiv:2602. 05970v2 Announce Type: replace-cross Abstract: Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies.

By Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore

llms safety

arXiv Machine Learning

Jul 31

Critical attention scaling in long-context transformers

arXiv:2510. 05554v2 Announce Type: replace Abstract: As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length $n$ increases, causing tokens to cluster excessively, a phenomenon known as rank-collapse.

By Shi Chen, Zhengjiang Lin, Yury Polyanskiy, Philippe Rigollet

llms

arXiv Machine Learning

Jul 30

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

arXiv:2505. 15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic.

By Suvadeep Hajra

llms

arXiv Machine Learning

Jun 25

Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.

By Yizhou Liu, Jeff Gore

llms

arXiv AI

Jun 8

Limitations of Normalization in Attention Mechanism

arXiv:2508. 17821v3 Announce Type: replace-cross Abstract: This paper investigates the limitations of the normalization in attention mechanisms.

By Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova, Radu State

llms

arXiv AI

Jun 2

Universal One-third Time Scaling in Learning Peaked Distributions

arXiv:2602. 03685v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable.

By Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore

llms

One email a morning, machine-written

Related stories

Inverse Depth Scaling From Most Layers Being Similar

Critical attention scaling in long-context transformers

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

Limitations of Normalization in Attention Mechanism

Universal One-third Time Scaling in Learning Peaked Distributions