arXiv Computation and Language November 2, 2026 By Leon Lufkin, Cutter Dawes, Alexander Speicher, Kamesh Krishnamurthy, Tom\'as Figliolia, Beren Millidge

Subquadratic Attention with a Hierarchical Tree-Structured Indexer

Read the original on arXiv Computation and Language →

The paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention mechanism that uses a tree-structured indexer to compress key-value pairs and retrieve a sparse subset of keys during query time. By iteratively deepening the tree, ZHA achieves sublinear routing and attention costs, resulting in overall subquadratic scaling for long-context processing. The method can be applied to existing pretrained models with minimal distillation training, maintains performance on diverse evaluation tasks, and remains robust even with very deep trees.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jun 29

Simplified Sparse Attention via Gist Tokens

arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.

By Yuzhen Mao, Michael Y. Li, Emily B. Fox

llms rag efficiency

arXiv AI

Jul 23

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.

By Yu Zhao, Zekun Zhang, Fan Jiang, Bo Zeng, Linlong Xu, Shimin Shan, Yu Liu, Longyue Wang, Weihua Luo

efficiency benchmarks

arXiv Machine Learning

Sep 29

Block Sparse Attention with Log-Linear Complexity

arXiv:2609. 31093v1 Announce Type: new Abstract: Scaling language models to long contexts is limited by the quadratic cost of self-attention.

By Bohao Tang, Zhen Qin, Yuqi Pan, Zheng Li, Pengfei Liu

llms benchmarks

arXiv AI

2d ago

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

arXiv:2610.42147v1 Announce Type: new Abstract: Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context wind...

By Ales Kubicek, Andrea Caciolai, Liangzu Peng, Sukrit Sumant, Benjamin Hoover, Fraser Greenlee, Ruairidh M. Battleday, Radim Urban, Torsten Hoefler, James C. R. Whittington

agents diffusion efficiency

arXiv AI

Oct 19

Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.

By Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su, Qingqing Dang, Yanlin Sha, Kaipeng Deng, Dianhai Yu, Yanjun Ma, Juntao Li, Min Zhang

llms efficiency benchmarks

arXiv AI

Jul 7

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.

By Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang

llms

One email a morning, machine-written

Related stories

Simplified Sparse Attention via Gist Tokens

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Block Sparse Attention with Log-Linear Complexity

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling