← Back to all news
arXiv Machine Learning By Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, Daniel Y. Fu

BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

Read the original on arXiv Machine Learning →

The paper introduces BeNice!, a two‑stage approach that enables KV‑shareability in looped neural architectures. First, stochastic depth during pretraining drives representations toward a common limit across recurrences; second, a post‑training broadcast of the final recurrence’s KV‑cache to earlier recurrences trains the model to produce compatible residuals while preserving quality. BeNice! achieves KV‑shareability without sacrificing validation perplexity and can be combined with other memory‑efficient techniques, yielding significant gains in cache hit rates, throughput, and latency in low‑memory, long‑context inference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 28

BellowsKV: One Latent for Many Layers, Expanded on Demand

BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.

By Jiamu Zhang, Hanjie Chen
llmsagents
More like this →