arXiv AI

Where Position Matters: Profiling-Guided Layer-Wise RoPE Allocation in Transformers

The paper introduces a profiling-guided method for allocating Rotary Position Embedding (RoPE) coverage across Transformer layers, addressing the non-uniform positional needs revealed by phase‑off and coverage interventions. By using early coverage‑benefit profiles, the approach assigns high or low coverage under a fixed budget, supports fixed‑width query/key representations, and enables physical rotary‑dimension allocation in Multi‑head Latent Attention (MLA). Normalized‑depth mapping allows small‑model measurements to inform larger‑model allocations before pretraining, and experiments show validation‑loss improvements over budget‑matched baselines while highlighting favorable quality–storage–compute trade‑offs.

arXiv Machine Learning
Sep 4

Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

The paper proposes a principled way to design hybrid transformer architectures that combine Full Attention (FA) and Linear Attention (LA). By introducing two intervention metrics—RoPE Frequency Importance Score (RFIS) and RoPE Positional Dependence (RPD)—the authors identify a clear taxonomy of retrieval and positional heads, defining a Global Positional Band (GPBand) that aligns with training-length positional scales. Using these insights, they build a Head‑wise Hybrid Architecture (HwH) that assigns FA to global retrieval and LA to local positional modeling, achieving strong language modeling, improved retrieval, and superior zero‑shot long‑context extrapolation compared to standard Transformers and other hybrids.

By Runlin Shi, Bojian Yin, Guoqi Li
arXiv AI
Jul 23

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.

By Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li
arXiv Machine Learning
Oct 16

RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

RoLaQ is an offline technique that jointly calibrates rotation and quantization range for 2‑bit key‑value caches in multi‑head latent attention models. By refining a shared rotation within subspaces defined by frozen key and value projections and learning coupled quantization thresholds and reconstruction gains, RoLaQ improves prediction matching and attention‑output supervision while keeping model weights frozen. Experiments on three MLA models and six benchmarks show that RoLaQ outperforms other 2‑bit methods, achieving higher average accuracy and, in a DeepSeek‑V2‑Lite implementation, nearly four times the BF16 KV capacity and more than double the decode throughput for 256K input tokens.

By Shiyu Ji, Peng Ling, Chen Lin, Shupeng Li, Yixuan Wang, Qingfu Zhu, Wanxiang Che
arXiv Machine Learning
Sep 2

Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective

The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.

By Ruoxi Yu, Haotian Jiang, Jingpu Cheng, Penghao Yu, Qianxiao Li, Zhong Li