arXiv Machine Learning June 3, 2026 By Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

Read the original on arXiv Machine Learning →

arXiv:2606. 03458v1 Announce Type: new Abstract: Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 16

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

arXiv:2505. 18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.

By Donghyun Son, Euntae Choi, Sungjoo Yoo

llms efficiency safety

arXiv Machine Learning

Jul 2

GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache

arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.

By Soosung Kim, Minjae Park, Eui-Young Chung, Jaeyong Chung

llms efficiency safety

arXiv AI

Jul 21

High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

arXiv:2607. 16248v1 Announce Type: cross Abstract: Long-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads.

By Gradwell Dzikanyanga, Yanqi Pan, Weihao Yang, Donglei Wu, Wen Xia, Hao Huang

llms efficiency

arXiv Machine Learning

Jul 3

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.

By Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

llms rag agents efficiency benchmarks

arXiv AI

Jun 17

AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.

By Ning Ni, Yingjie Lao

llms efficiency safety

arXiv AI

Aug 6

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

arXiv:2608. 04048v1 Announce Type: cross Abstract: Serving large language models (LLMs) under diverse deployment constraints requires flexible trade-offs between accuracy, memory footprint, and throughput.

By Yu Luo, Bo Dong, Wenhua Cheng, Haihao Shen

llms efficiency

Related stories

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache

High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs