← Back to all news
arXiv Machine Learning By Kaicheng Xiao, Liran Dong, Haotian Li, Guoliang Xing

Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration

Read the original on arXiv Machine Learning →

The paper introduces RAM‑Net, a mechanism that unifies KV‑cache quantization and linear attention by using soft assignments over a discrete address space to update continuous slot states. It shows that these soft assignments extend hard quantized matching to a separable read‑write overlap, locally approximating full‑attention similarity while enabling recurrent aggregation. The authors demonstrate that RAM‑Net can migrate pretrained Transformer weights and recover an average of 87.1% of the original accuracy across nine models using only a 500M‑token budget.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
4d ago

One Distribution, Two Channels: Coverage-Calibrated KV Retention and Compensation

The paper introduces CORE, a method that jointly calibrates coverage and redistributes evicted mass for key–value (KV) cache in long‑context decoding. CORE combines query utility and log‑determinant coverage into a lightweight indexer, using a single calibrated distribution to both retain complementary KV states and parameterize latent‑memory writes. Experiments on three backbones show CORE outperforms the RULER baseline by up to 3.78 points at 90% compression and demonstrates strong decoding efficiency on LongBench and repeated‑eviction tests.

By Shuxin Liu, Qing Liu, Yi Du, Ou Wu
efficiency
More like this →