arXiv AI
Jun 9
End-to-End Context Compression at Scale
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models
The paper surveys memory‑centric compression techniques for large language models, framing long‑context handling as a memory‑management problem. It distinguishes compute‑active memory from compute‑inactive memory, and models operations such as writing, offloading, recall, and forgetting as state transitions within an information lifecycle. The authors propose a unified cost model and argue that future research should focus on dynamic, lifecycle‑aware memory management rather than static compression.
Related stories
arXiv AI
Jun 24
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
arXiv Machine Learning
Jul 10
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.
Hugging Face Trending Papers
Jun 8
End-to-End Context Compression at Scale
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.
arXiv AI
Sep 30
KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...
arXiv Machine Learning
Oct 8
KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
arXiv:2610.08811v1 Announce Type: new Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...
arXiv AI
Aug 18
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
Hugging Face Trending Papers
Jun 23
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv Machine Learning
Sep 1
What It Costs to Compose, Rebuild, and Correct Precomputed Memory
arXiv:2608.30647v1 Announce Type: cross Abstract: Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at ea...
arXiv Machine Learning
Sep 10
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
arXiv Computation and Language
Sep 14
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
arXiv:2608.00311v2 Announce Type: replace Abstract: Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and th...
arXiv AI
Jun 30
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.