The paper introduces Semantic Head Specialization (SHS), a phenomenon where Vision Transformer (ViT) attention heads specialize as either object- or background-focused, most evident under full attention. It proposes the SHS-Index to quantify this specialization, demonstrating its ability to distinguish full-attention from chunk-window ViTs and its strong correlation with downstream benchmark performance. Leveraging insights into window interaction, token serialization, and local softmax allocation, the authors design Ariadne Attention, a hybrid attention mechanism that matches full-attention performance on 22 image and video tasks while reducing attention compute by 6.5×.
By Chenhong He, Lei Li, Shicheng Li, Hanglong Lv, Lingpeng Kong, Qi Liu, Tong Yang, Shuhuai Ren
The paper introduces GatedV, an attention variant that applies the SiLU activation only to the value matrix before aggregation, adding no extra trainable parameters. Experiments on a decoder‑only Transformer show that, with matched parameter counts and training budgets, GatedV consistently lowers held‑out perplexity—by 2.0% at 2.33M parameters and 2.4% at 26.16M parameters—across multiple seeds. Ablations indicate that the performance gains are not fully captured by simple affine rescaling of values, and a benchmark on NVIDIA DGX Spark reports a modest 0.7% increase in full‑context forward latency for the larger model.
By Li-Ming Yu, Hongyu Zhou
MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.
By Prasoon Dev, Anirudh Sankar, Vasudeva Varma
arXiv:2507. 07247v2 Announce Type: replace-cross Abstract: As large language models (LLMs) and visual language models (VLMs) grow in scale and application, attention mechanisms have become a central computational bottleneck due to their high memory and time complexity.
By Zhengyu Tian, Anantha Padmanaban Krishna Kumar, Hemant Krishnakumar, Reza Rawassizadeh
The paper introduces Multi-Stream Attention Residuals (MSAR), an extension of Attention Residuals that applies multi-head depth attention inspired by multi-head sequence attention. MSAR uses multi-scale causal convolutions to provide diverse contextual views for each head, then adaptively mixes the retrieved streams into a single-stream input, avoiding persistent multi-stream storage. Across 12 downstream benchmarks at 18B and 28B MoE scales, MSAR outperforms prior residual variants and achieves a 1.58× compute speedup over the standard residual baseline.
By Xiaohan Qin, Xiangdong Zhang, Tuo Dai, Sunan Zou, Huaijin Wu, Yebin Yang, Zhuo Xia, Lin Yao, Yuliang Liu, Junchi Yan
Kairos is a byte‑level foundation model that integrates bidirectional gated‑DeltaNet linear attention, sliding‑window attention, block‑wise Attention‑Residual aggregation, a sparse Mixture‑of‑Experts feed‑forward layer, and a multi‑scale byte‑level codec. It is trained with a uniform‑noise denoising objective over a single byte space shared by text, image, audio, video, lidar, and control streams, and the authors provide code‑derived algorithms for each component, including a cross‑session recurrent‑state memory gate. Convergence sanity checks on a 440‑byte text sentence show that the two attention variants reach a minimum loss near 10⁻³, and the implementation appears to optimize, though generalization and multimodal performance remain to be evaluated in future pretraining runs.
By Fabien Furfaro
Multi-Stream Attention Residuals (MSAR) extends the Attention Residuals (AttnRes) mechanism by introducing multi-head depth attention, inspired by multi-head attention along the sequence axis. MSAR generates diverse streams from sequence context via multi-scale causal convolutions, then adaptively mixes them to produce a single-stream layer input, enabling multi-stream computation without extra storage or communication. Across 12 downstream benchmarks at 18B and 28B MoE scales, MSAR consistently outperforms prior residual variants and achieves a 1.58× iso-loss compute speedup over the standard residual baseline.
arXiv:2608.30320v1 Announce Type: new
Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
By Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu
arXiv:2605. 18848v3 Announce Type: replace Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error.
By Weinuo Ou
RAVE (Re-Allocating Visual Attention) is a lightweight pair‑gating mechanism that adds a learned query‑key bias to pre‑softmax attention scores over visual keys, derived from pre‑RoPE query and key features. It requires no architectural changes to the backbone and can be trained end‑to‑end with the rest of the model. Across multiple multimodal benchmarks, RAVE improves standard attention by an average of 3 points, especially on perception‑intensive tasks such as multilingual OCR, chart understanding, document VQA, and scene text VQA.
By Xi Leng, Xinhong Ma, Ziqiang Dong, Feng Zhang, Xiaoying Tang, Yang Yang, Guanjun Jiang
The paper examines why learned gates in sparse attention models offer little advantage over random gates when jointly trained with the transformer. Through experiments on a 31M-parameter transformer, the authors attribute this to routing absorption, where the model’s representations adapt to the imposed mask, diminishing the benefit of learned routing. They also explore hard masking, stochastic mask training, and the impact of trainable attention layers on gate performance, concluding that freezing the model stabilizes routing targets for effective post‑hoc sparsification.
By Keston Aquino-Michaels
arXiv:2609.26368v1 Announce Type: new
Abstract: Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context dem...
By Jianyu Wei, Yizhao Gao, Qihao Zhang, Shimao Chen, Zhengju Tang, Yu Cheng, Shengjie Zhou, Zihan Jiang, Yifan Song, Hailin Zhang, Liang Zhao, Bo Yang, Gang Wang, Shijie Cao, Fuli Luo