← Back to all news
arXiv Machine Learning By Adam Deng, Shicheng Liu, Yada Zhu

Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches

Read the original on arXiv Machine Learning →

The paper investigates whether a frozen transformer can use another model’s KV cache as a substitute for its own, finding that this approach fails because the reader only accepts caches within a narrow tolerable set. Experimental results show that none of eight substitution methods improve performance over a baseline, while a different strategy—adding the sender’s cache as extra context and only adapting queries—does outperform the baseline. The authors conclude that pursuing faithful cache substitution is misguided and that augmenting the reader’s own cache is the more promising direction.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 10

What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.

By Ashwin Gerard Colaco, Nada Lahjouji
llmsagentsefficiencybenchmarks
More like this →
arXiv Machine Learning
Oct 16

Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing

The paper investigates how to efficiently share internal states between agents built from frozen language models. It proposes a shared pool approach where each agent publishes its internal state once, allowing peers to read it via a single mechanism, thereby avoiding the need for a dedicated transfer bridge for every agent pair. The study finds that augmenting the key‑value cache and replacing activations—rather than swapping them—yields better accuracy with minimal training, and that a single adapter can generalize across multiple publishers when their internal states are similar to the receiver’s.

By Adam Deng, Shicheng Liu, Yada Zhu
llmsagentsefficiency
More like this →
arXiv Computation and Language
Sep 4

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. Experiments on four models and six reasoning tasks show that this simple strategy matches the performance of the best existing eviction methods and achieves 32‑43% higher throughput in vLLM deployments. The authors explain that the prompt is the most fragile cache component and that reasoning traces are redundantly stored across text and attention heads, making a selection score unnecessary.

By Heng Wang, Jielin Qiu, Wenting Zhao, Cheng Qian, Liangwei Yang, Jiawei Han, Heng Ji, Silvio Savarese, Shelby Heinecke, Huan Wang
llmsefficiency
More like this →