The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2609.28029v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and groupe...
arXiv:2607.12550v3 Announce Type: replace-cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, a...
arXiv:2607. 12550v1 Announce Type: new Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference.
Activation-Aware Weight Tensorization (AWT) is a training‑free calibration wrapper that preconditions each weight matrix with a diagonal activation‑derived scale before applying standard Tensor Train (TT) or Tree Tensor Network (TTN) decompositions. By adding only an input‑side elementwise rescaling, AWT improves functional fidelity of post‑training tensor‑network compression across several large language models (Llama 3.1 8B, Ministral 8B, Qwen2.5 7B) at 2–6× compression, closing a significant portion of the perplexity gap to dense baselines and enhancing downstream task performance. The method demonstrates that diagonal preconditioning offers a robustness‑modularity trade‑off, outperforming full‑covariance oracles in many cases while remaining modular and solver‑agnostic.
The paper introduces JoLT, a training‑free compressor that jointly allocates rank and precision for key‑value (KV) cache compression in long‑context language models. JoLT treats grouped prefill caches as fourth‑order tensors, applies partial Tucker decomposition along token and feature modes, and uses a rotated low‑bit quantizer for residuals, all governed by a single Lagrangian dual under a global byte constraint. Across five models from four architecture families, JoLT achieves 2–3× compression with less than 0.2% perplexity loss, and near‑lossless retrieval accuracy on LLaMA‑3.1‑8B at 64K context up to 3× compression.
arXiv:2606. 03465v1 Announce Type: cross Abstract: Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints.