arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
By Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh, Tan Lai Ngoc, Tan M. Nguyen
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
By Shubham Aggarwal
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
By Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li
arXiv:2609.38109v1 Announce Type: cross
Abstract: The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of...
By Cutter Dawes, Nick Alonso, Tom Figliolia, Beren Millidge
arXiv:2610.07848v1 Announce Type: cross
Abstract: Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing...
By Dayan Pan, Jingyuan Wang, Xie Yu
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
By Subham Singh, Ashutosh Mishra, Subha Raut
arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.
By Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee
arXiv:2609.38530v1 Announce Type: new
Abstract: Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-windo...
By Eric Enouen, Sainyam Galhotra
arXiv:2609.01231v1 Announce Type: new
Abstract: We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to t...
By W. Ross Morrow
arXiv:2610.16301v1 Announce Type: new
Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...
By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)
The paper introduces a profiling-guided method for allocating Rotary Position Embedding (RoPE) coverage across Transformer layers, addressing the non-uniform positional needs revealed by phase‑off and coverage interventions. By using early coverage‑benefit profiles, the approach assigns high or low coverage under a fixed budget, supports fixed‑width query/key representations, and enables physical rotary‑dimension allocation in Multi‑head Latent Attention (MLA). Normalized‑depth mapping allows small‑model measurements to inform larger‑model allocations before pretraining, and experiments show validation‑loss improvements over budget‑matched baselines while highlighting favorable quality–storage–compute trade‑offs.
By Zhiwei Liu, Boao Kong, Bing Liu
arXiv:2605. 24059v2 Announce Type: replace Abstract: We present a three-step recipe for identifying attention-head circuits in pretrained transformers.
By Yongzhong Xu