arXiv Machine Learning July 1, 2026 By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Read the original on arXiv Machine Learning →

arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 5

Output-Aware Rotation for INT2 KV-Cache Quantization

arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.

By Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong, Sunwoo Lee, Murali Annavaram, Sai Praneeth Karimireddy, Sungjoo Yoo

llms efficiency benchmarks

arXiv Machine Learning

Jul 3

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.

By Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

llms rag agents efficiency benchmarks

arXiv AI

Jun 9

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

arXiv:2606. 09508v1 Announce Type: new Abstract: Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts.

By Zhanchao Xu, Haoyang Li, Qingfa Xiao, Fei Teng, Chen Jason Zhang, Lei Chen, Qing Li

llms efficiency

arXiv Machine Learning

Jul 2

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

arXiv:2508. 18224v3 Announce Type: replace-cross Abstract: Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language models (LLMs).

By Ran Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai, Beidi Chen, Binhang Yuan

llms benchmarks

arXiv AI

Jun 4

Stochastic Sparse Attention for Memory-Bound Inference

arXiv:2605. 01910v2 Announce Type: replace-cross Abstract: Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache.

By Kyle Lee, Corentin Delacour, Kevin Callahan-Coray, Kyle Jiang, Can Yaras, Samet Oymak, Tathagata Srimani, Kerem Y. Camsari

llms efficiency

arXiv AI

Jul 31

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

arXiv:2606. 21848v2 Announce Type: replace-cross Abstract: Transformer architectures form the foundation of modern natural language processing, making it crucial to address the efficiency and scalability limitations of the standard QKV attention mechanism.

By Xin Gao, Xingming Xu

llms nlp efficiency benchmarks

Related stories

Output-Aware Rotation for INT2 KV-Cache Quantization

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

Stochastic Sparse Attention for Memory-Bound Inference

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers