arXiv Machine Learning

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution

The paper introduces QUILT, a new sparse‑attention execution mechanism that jointly processes neighboring queries to reuse shared key‑value (KV) entries, thereby reducing redundant memory traffic and computation. QUILT employs Shift‑and‑Compare Set Decomposition (SCSD) to convert irregular set operations into regular data‑parallel primitives and pipelines this with attention computation to hide overhead. The approach includes cascaded sharing across multiple granularities and a tile‑aware strategy that balances sharing with hardware utilization, achieving up to 55.1% lower kernel latency and 36.8% faster time‑to‑first‑token on LongBench benchmarks with negligible accuracy loss.

arXiv Machine Learning

5d ago

PageWeaver: KV-Guided Query Unions for Sparse Attention

PageWeaver introduces a KV‑guided query union strategy for sparse attention, assembling query groups based on selected‑page affinity while preserving each query’s support and output ownership. The design employs a bounded GPU search to generate query IDs and an ID‑aware two‑CTA kernel that consumes them without materializing reordered Q tensors or cross‑page partial outputs. Experiments on the H200 Union8 with FP8 KV show a 1.70× speedup over FlashInfer, with additional latency reductions of 3.26–7.66% on selected captures and modest throughput gains for whole‑model prefill.

By Zhiyuan Li, Zihan Li, Zefang Yuan, Lei Wang, Hao Wang

efficiency

arXiv Machine Learning

Jun 3

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.

By Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao, Bin Cui

llms rag agents efficiency

arXiv AI

Jun 9

FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.

By Yan Wang, Qifan Zhang, Jiachen Yu, Tian Liang, Dongyang Ma, Xiang Hu, Zibo Lin, Chunyang Li, Zhichao Wang, Jia Li, Yujiu Yang, Haitao Mi, Dong Yu

llms efficiency

arXiv Machine Learning

Jun 5

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.

By Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei

llms efficiency benchmarks

arXiv AI

Jun 12

MiniMax Sparse Attention

arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.

By Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao

llms agents efficiency multimodal

Hugging Face Trending Papers

Jun 25

PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication. Modern paged-attention systems reduce KV-cache fragmentation and mature kernels such as FlashInfer provide highly optimized native-paged decode attention.

llms efficiency

arXiv Machine Learning

Jun 4

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.

By Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa

llms efficiency

arXiv Machine Learning

Jun 26

PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

arXiv:2606. 26666v1 Announce Type: new Abstract: Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication.

By Muhammad Ahmed

llms efficiency

arXiv AI

Aug 10

Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.

By Sanjeev Rao Ganjihal

agents efficiency

arXiv Machine Learning

Jul 27

HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

arXiv:2607. 22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck.

By Chao Fang, Jun Yin, Man Shi, Marian Verhelst

llms efficiency benchmarks

arXiv Machine Learning

Oct 2

SparseEngine: Sparse-First Inference Engine

arXiv:2609.39068v1 Announce Type: new Abstract: Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these...

By Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu

llms agents efficiency benchmarks

arXiv AI

2d ago

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

arXiv:2507.19608v2 Announce Type: replace Abstract: On-device LLM inference is increasingly important for edge platforms but remains challenging due to high computational and memory demands and tight...

By Jiawen Qi, Jiayu Liu, Chang Gao, Zhaochun Ren, Qinyu Chen

llms fine-tuning efficiency

Related stories