Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

Kronecker Product Attention (KPA) is introduced as a memory‑efficient method for compressing the key‑value cache in autoregressive models. By representing key and value activations as sums of Kronecker products of compact, token‑dependent matrix factors, KPA generalizes Tensor Product Attention and can capture richer block‑structured patterns. Empirical results show that KPA matches or surpasses existing techniques such as MHA, MQA, GQA, and MLA across multiple model scales and benchmarks, achieving up to 10× compression while preserving relative‑position information through a structured rotary embedding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• rag
• efficiency
• benchmarks