arXiv Machine Learning July 1, 2026 By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Read the original on arXiv Machine Learning →

arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 5

Output-Aware Rotation for INT2 KV-Cache Quantization

arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.

By Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong, Sunwoo Lee, Murali Annavaram, Sai Praneeth Karimireddy, Sungjoo Yoo

llms efficiency benchmarks

arXiv Computation and Language

Aug 21

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

arXiv:2608. 19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets.

By Matthias Seeger, Zeyu Zhang, Vihang Patil, Konstantinos Benidis, Sebastian Schelter

llms fine-tuning efficiency

arXiv Machine Learning

Jul 3

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.

By Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

llms rag agents efficiency benchmarks

Hugging Face Trending Papers

Aug 20

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention.

llms fine-tuning efficiency

arXiv AI

Sep 17

HyQuant: Hybrid-Precision Quantization for LLM Attention

HyQuant introduces a hybrid-precision quantization framework for large language model (LLM) attention modules. It quantizes most attention states to low-bit formats while preserving a small set of vertical‑line tokens and local‑window states in full precision, guided by lightweight attention‑pattern signals. This design achieves near‑lossless accuracy across tasks while improving memory and hardware efficiency.

By Jiatong Ding, Bingxin Xing, Yu Zhang, Dian Ding, Xiaodong Yi, Xianbin Ouyang, Feihu Zhou, Kun Zhang, Zhenyu Guo, Hao Pan, Guangtao Xue, Yiming Zhang

llms efficiency

arXiv AI

Sep 1

CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

arXiv:2608.30295v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challeng...

By Haoyun Jiang, Haolin Li, Jianwei Zhang, Fei Huang, Qiang Hu, Minmin Sun, Shuai Xiao, Yong Li, Junyang Lin, Jiangchao Yao

llms efficiency benchmarks

One email a morning, machine-written

Related stories

Output-Aware Rotation for INT2 KV-Cache Quantization

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

HyQuant: Hybrid-Precision Quantization for LLM Attention

CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration