FlyWheel introduces optimized TPU kernels for softmax and linear attention, reducing vector work and improving matrix operand reuse. By overlapping independent computations, FlyWheel achieves up to 4.14× speedups over the RPA kernel and 1.80× over Splash Attention, reaching 81% of peak BF16 throughput on TPU v6e. It also boosts Gated DeltaNet prefill by up to 3.61× and enhances vLLM serving of Qwen3.8-27B prefill throughput by up to 1.56× while preserving accuracy.
arXiv:2611.06828v1 Announce Type: new Abstract: Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core til...
FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
arXiv:2610.31911v1 Announce Type: new Abstract: As large language models (LLMs) process increasingly long sequences, attention becomes a major computational bottleneck. Sparse attention mostly target...
arXiv:2607. 07953v1 Announce Type: cross Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context.
arXiv:2608.30320v1 Announce Type: new Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
The paper investigates using short polynomial approximations to accelerate special‑function operations in large language models on NVIDIA Blackwell GPUs. By replacing native sigmoid, tanh, and SiLU with degree‑3 or degree‑4 bfloat16 programs, the authors achieve up to 2.19× speed‑ups in isolated FP16 benchmarks and modest training‑step throughput gains (2.7–8.0%) across four integration tasks. The study also evaluates model behavior, finding negligible training‑loss differences within 100 billion tokens.
arXiv:2606. 06453v1 Announce Type: new Abstract: Sparse attention is becoming increasingly important for serving large language models (LLMs) as generation lengths continue to grow.
arXiv:2609.13612v1 Announce Type: new Abstract: Modern AI systems are built on the Transformer architecture, whose core operation, attention, accounts for the majority of computation and memory cost....
arXiv:2508. 18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs).
DeltaLLM is a training‑free framework that accelerates on‑device large language model inference by introducing delta‑key sparse attention and a vectorized structured sparse online softmax. It reuses delta keys across query heads, preserves exact attention near the diagonal while approximating non‑local regions, and exploits hardware‑friendly sparsity with a specialized vector‑processor architecture. Experiments on BitNet‑b1.58‑2B‑4T and LLaMA3.2‑1B‑Instruct show up to 60% delta‑key sparsity, 63% structured attention‑score sparsity, and a 1.62× speedup on the KV260 FPGA with minimal accuracy loss.
Kairos is a byte‑level foundation model that integrates bidirectional gated‑DeltaNet linear attention, sliding‑window attention, block‑wise Attention‑Residual aggregation, a sparse Mixture‑of‑Experts feed‑forward layer, and a multi‑scale byte‑level codec. It is trained with a uniform‑noise denoising objective over a single byte space shared by text, image, audio, video, lidar, and control streams, and the authors provide code‑derived algorithms for each component, including a cross‑session recurrent‑state memory gate. Convergence sanity checks on a 440‑byte text sentence show that the two attention variants reach a minimum loss near 10⁻³, and the implementation appears to optimize, though generalization and multimodal performance remain to be evaluated in future pretraining runs.