arXiv Machine Learning

Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

The paper proposes a principled way to design hybrid transformer architectures that combine Full Attention (FA) and Linear Attention (LA). By introducing two intervention metrics—RoPE Frequency Importance Score (RFIS) and RoPE Positional Dependence (RPD)—the authors identify a clear taxonomy of retrieval and positional heads, defining a Global Positional Band (GPBand) that aligns with training-length positional scales. Using these insights, they build a Head‑wise Hybrid Architecture (HwH) that assigns FA to global retrieval and LA to local positional modeling, achieving strong language modeling, improved retrieval, and superior zero‑shot long‑context extrapolation compared to standard Transformers and other hybrids.

arXiv AI
Jul 23

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.

By Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li
arXiv AI
Jul 22

A Controlled Study of Attention-Only Transformers

arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.

By Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee
arXiv AI
Oct 15

FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

arXiv:2610.16301v1 Announce Type: new Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...

By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)
arXiv AI
Oct 31

Where Position Matters: Profiling-Guided Layer-Wise RoPE Allocation in Transformers

The paper introduces a profiling-guided method for allocating Rotary Position Embedding (RoPE) coverage across Transformer layers, addressing the non-uniform positional needs revealed by phase‑off and coverage interventions. By using early coverage‑benefit profiles, the approach assigns high or low coverage under a fixed budget, supports fixed‑width query/key representations, and enables physical rotary‑dimension allocation in Multi‑head Latent Attention (MLA). Normalized‑depth mapping allows small‑model measurements to inform larger‑model allocations before pretraining, and experiments show validation‑loss improvements over budget‑matched baselines while highlighting favorable quality–storage–compute trade‑offs.

By Zhiwei Liu, Boao Kong, Bing Liu