← Back to all news
arXiv Machine Learning By Gil Jae Song, Sungjoo Yoo

PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference

Read the original on arXiv Machine Learning →

PN-Mix is a mixed‑precision key‑cache quantization technique for RoPE‑based decoder‑only large language models that allocates higher precision to salient channel pairs identified during the prefill stage. By focusing on channel pairs with large query–key pair‑norm products, PN‑Mix reduces logit variation caused by quantization errors. Experiments on reasoning, code‑generation, and AIME benchmarks show that PN‑Mix achieves comparable or better accuracy than prior methods while using an average precision of 2.26 bits and degrades gracefully down to 1.5 bits.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 8

Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches

Dual-QK introduces paired non‑orthogonal transforms for queries and keys to enable sharp queries and flat keys, improving 2‑bit quantization and dynamic channel pruning of KV caches. The method balances key scales for INT2 quantization while concentrating query energy for pruning, and supports channel‑0 protection and bucket‑relative RoPE for long‑context accuracy. Experiments on four models across five generative benchmarks show that Dual‑QK outperforms OSCAR at 40% query‑channel sparsity, achieving 6.8× KV‑cache compression, an 8.3× reduction in KV read volume, and up to 3.75× decoding throughput over unpruned BF16.

By Sunjoo Whang, Jungjun Oh, Minsung Kim, Dongho Seo, Jisu Shin, Gregory Kielian, Hoi-Jun Yoo, Sangjin Kim
efficiencybenchmarks
More like this →