arXiv AI
Jun 9
End-to-End Context Compression at Scale
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
PReM: Learning What to Preserve and When to Refresh for Context Compression
arXiv:2607. 14327v1 Announce Type: cross Abstract: Efficient long-context inference is not only about reducing memory cost, but also about keeping useful contextual evidence accessible as generation proceeds.
Related stories
Hugging Face Trending Papers
Jun 8
End-to-End Context Compression at Scale
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.
arXiv AI
Jul 8
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
arXiv Machine Learning
5d ago
MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.
arXiv AI
Jul 3
Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v1 Announce Type: cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv AI
1d ago
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
arXiv AI
Jul 7
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v2 Announce Type: replace-cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv AI
Jun 3
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
arXiv:2605. 03644v2 Announce Type: replace Abstract: Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs).
arXiv AI
Jun 24
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
arXiv AI
Jul 8
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.
arXiv Machine Learning
Aug 4
LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference
arXiv:2608. 02515v1 Announce Type: cross Abstract: Long-running assistants and agents consume interaction streams that eventually outgrow the context.
arXiv Machine Learning
Aug 4
Learning What to Remember: Test-Time Training via Context Distillation
arXiv:2608. 01672v1 Announce Type: cross Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later.