← Back to all news
arXiv Machine Learning By Shikhar Srivastava, Christopher Kanan

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

Read the original on arXiv Machine Learning →

LayerRoPE introduces a depth‑wise magnitude and angular superposition mechanism that replaces per‑layer normalization gains with a single shared vector and depth‑conditioned scalars, reducing parameters and computational cost. Across 16 pre‑trained large language models and a 100B+ token model ladder, LayerRoPE outperforms traditional Pre‑, Post‑, and Peri‑Norm schemes, achieving comparable or better performance with significantly less compute and improved convergence up to 512 layers. The method demonstrates that widening the residual stream, rather than shrinking it, enhances depth stability and learning‑rate sensitivity, and it transfers effectively to latent and vision transformer architectures.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Jul 15

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.

llms
More like this →
arXiv Machine Learning
Sep 24

The Drift Contract: Spectral Updates for Depth-Robust Local Learning

The paper introduces the Drift Contract, a spectral update geometry for local learning that improves depth robustness and hyperparameter stability. By applying momentum orthogonalization with spectral step scaling to per‑layer updates, the authors achieve consistent performance across a wide range of widths and depths on CIFAR‑10 MLPs, outperforming local Adam and providing a per‑layer, input‑conditioned drift bound. The study also shows that the spectral geometry itself, rather than step‑size rules, drives the observed depth robustness, while a negative result indicates that the stability benefit is limited to non‑normalized layers.

By Fabien Polly
benchmarks
More like this →