The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.
arXiv:2605. 15250v3 Announce Type: replace-cross Abstract: Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly.
arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.
Virtual Head Attention (VHA) is a lightweight attention mechanism that improves long‑context large language models by adding two linear operations—Q Premix and Linear Postmix—to recover head diversity and fuse inter‑head features without nonlinearities. VHA reduces KV cache size, training and prefill FLOPs, and maintains or surpasses model quality compared to existing methods such as GQA, MLA, and MFA across multiple scales and benchmarks. The design allows VHA to be folded into standard GQA inference pipelines, enabling efficient deployment with only two KV heads and full exploitation of Tensor Core parallelism.
PageWeaver introduces a KV‑guided query union strategy for sparse attention, assembling query groups based on selected‑page affinity while preserving each query’s support and output ownership. The design employs a bounded GPU search to generate query IDs and an ID‑aware two‑CTA kernel that consumes them without materializing reordered Q tensors or cross‑page partial outputs. Experiments on the H200 Union8 with FP8 KV show a 1.70× speedup over FlashInfer, with additional latency reductions of 3.26–7.66% on selected captures and modest throughput gains for whole‑model prefill.
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.
The paper introduces QUILT, a new sparse‑attention execution mechanism that jointly processes neighboring queries to reuse shared key‑value (KV) entries, thereby reducing redundant memory traffic and computation. QUILT employs Shift‑and‑Compare Set Decomposition (SCSD) to convert irregular set operations into regular data‑parallel primitives and pipelines this with attention computation to hide overhead. The approach includes cascaded sharing across multiple granularities and a tile‑aware strategy that balances sharing with hardware utilization, achieving up to 55.1% lower kernel latency and 36.8% faster time‑to‑first‑token on LongBench benchmarks with negligible accuracy loss.
arXiv:2606. 30709v1 Announce Type: cross Abstract: Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers.
arXiv:2606. 01502v1 Announce Type: cross Abstract: Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk.
arXiv:2610.07940v1 Announce Type: cross Abstract: Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key...