arXiv Computation and Language November 4, 2026 By Zishuo Bao, Zeshen Zhang, Linkun Chen, Yucheng Lu

FlyWheel: Attention Kernels Built for the TPU

Read the original on arXiv Computation and Language →

FlyWheel introduces optimized TPU kernels for softmax and linear attention, reducing vector work and improving matrix operand reuse. By overlapping independent computations, FlyWheel achieves up to 4.14× speedups over the RPA kernel and 1.80× over Splash Attention, reaching 81% of peak BF16 throughput on TPU v6e. It also boosts Gated DeltaNet prefill by up to 3.61× and enhances vLLM serving of Qwen3.8-27B prefill throughput by up to 1.56× while preserving accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

1d ago

Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs

arXiv:2611.06828v1 Announce Type: new Abstract: Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core til...

By Gagik Amirkhanyan, Nuojin Cheng

arXiv Machine Learning

Sep 22

FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.

By Anthony Givans, Michael Crawshaw, Mingrui Liu

llms

arXiv AI

Jun 12

MiniMax Sparse Attention

arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.

By Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao

llms agents efficiency multimodal

arXiv Computation and Language

Oct 26

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning

arXiv:2610.31911v1 Announce Type: new Abstract: As large language models (LLMs) process increasingly long sequences, attention becomes a major computational bottleneck. Sparse attention mostly target...

By Keqi Deng, Ruchao Fan, Jinyu Li

llms fine-tuning efficiency

arXiv AI

Jul 10

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

arXiv:2607. 07953v1 Announce Type: cross Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context.

By Tommaso Cerruti, Tim Rieder, George Rowlands, Lingfeng Jin, Imanol Schlag

efficiency benchmarks

arXiv Computation and Language

Sep 1

On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

arXiv:2608.30320v1 Announce Type: new Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...

By Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu

rag benchmarks