← Back to all news
arXiv AI By Prasoon Dev, Anirudh Sankar, Vasudeva Varma

MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution

Read the original on arXiv AI →

MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 10

Dynamic Linear Attention

arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.

By Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang
llmsbenchmarks
More like this →
Hugging Face Trending Papers
Jun 9

Dynamic Linear Attention

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner.

llmsbenchmarks
More like this →
arXiv Machine Learning
Aug 31

InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model

InfoMamba is an attention‑free hybrid model that combines a minimal‑bandwidth global interface with a selective recurrent stream. The architecture replaces token‑level self‑attention with a concept bottleneck linear filtering layer and integrates it via an information‑maximizing fusion (IMF) that injects global context into the state‑space dynamics. Experiments across classification, dense prediction, and non‑vision tasks show that InfoMamba outperforms strong Transformer and SSM baselines while maintaining near‑linear scaling and competitive accuracy‑efficiency trade‑offs.

By Youjin Wang, Jiaqiao Zhao, Rong Fu, Run Zhou, Ruizhe Zhang, Jiani Liang, Suisuai Cao, Feng Zhou
llms
More like this →