The paper investigates how to recover language model quality lost when using low‑bit key–value caches for autoregressive decoding. By keeping the quantizer fixed and distilling the full‑precision cache behavior into low‑rank Q/K/V projection updates, the authors demonstrate that 4‑bit affine‑cache adapters recover roughly 54 % of the perplexity gap on TinyLlama‑1.1B and 76 % on Gemma‑4‑12B, while preserving most long‑context retrieval. Even a 2‑bit rank–token sweep can dramatically reduce TinyLlama’s perplexity, though it only partially restores retrieval performance.
By Seifeldin Abdellatif
arXiv:2609.26333v1 Announce Type: new
Abstract: Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce me...
By Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh
arXiv:2608.30427v1 Announce Type: cross
Abstract: Speculative decoding speeds up generation with an efficient draft model (drafter) that proposes tokens for a target model to verify in one pass, pres...
By Ephrem Wu
The paper investigates whether training Mixture-of-Experts (MoE) routers can improve memory‑bandwidth locality on consumer GPUs. Using a new zero‑surgery telemetry tool, the authors measure that a large Qwen3‑235B model is bottlenecked by disk‑based expert access, and that an LRU cache can serve a majority of requests. They pre‑register experiments training 137 M‑parameter MoE models with locality‑aware losses, finding that while cache misses can drop up to 60 % (99 % static‑pin hit rate), every configuration fails to meet a strict 1 % perplexity threshold, indicating a tight coupling between cache efficiency and model quality.
By Shriniwas Ramesh Suram
arXiv:2609.17109v1 Announce Type: new
Abstract: A common small-model deployment runs one shared backbone with several LoRA specialists that answer over the same context. Serving them naively re-prefi...
By Dushyant Rajput
VestigeKV is a new KV‑cache technique that uses a 64‑dimensional vestigial branch—originally a RoPE component repurposed during NoPE training—as a query‑independent eviction signal. By reading only 11 % of each cache row, the method partitions the cache into an attended tier (top‑m rows) and an archive tier (all other rows), which is GPU‑resident and never deleted. The approach achieves near‑perfect retrieval (1.00 at 8×, 0.92 at 32×) without any training, quantization, or changes to weights or kernels.
By WenJie Fan