← Back to all news
arXiv Machine Learning By Xinglang Xian

Shared Global KV with Layer-Specific Local History

Read the original on arXiv Machine Learning →

The paper investigates how to combine shared global key‑value (KV) caches with layer‑specific local history in decoder‑only Transformer language models. By separating historical content from the input source used to form it, the authors show that adding local history can reduce held‑out test perplexity by about 1.4% compared to a current‑token local branch, while also demonstrating benefits in capacity, entry‑count, and training‑compute controls. Experiments on a 126M‑parameter model with 2K context reveal that local history remains valuable even when adjacent layers share local inputs, and that a sufficient suffix schedule can reduce upper‑layer construction work without losing cache completeness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 8

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.

By Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jesus Olivera
llmsnlpefficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 17

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.

By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
llmsdiffusionefficiency
More like this →