arXiv AI
Jun 9
End-to-End Context Compression at Scale
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Related stories
Hugging Face Trending Papers
Jun 8
End-to-End Context Compression at Scale
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.
arXiv Computation and Language
Sep 14
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
arXiv:2608.00311v2 Announce Type: replace Abstract: Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and th...
arXiv AI
Jun 30
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.
arXiv AI
Jun 24
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
arXiv AI
Jul 8
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
arXiv Computation and Language
Oct 15
Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
arXiv AI
Jun 2
Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.
Hugging Face Trending Papers
Aug 24
ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation
The quadratic growth of attention computation and key-value (KV) cache with respect to sequence length is a central bottleneck for ultra-long-context language models and high-resolution generative mod...
arXiv AI
Jul 31
ARC-Encoder: learning compressed text representations for large language models
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
arXiv Computation and Language
Oct 27
Linear-Time Context Compression with State-Space Encoding
arXiv:2610.33006v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compr...
arXiv Computation and Language
Sep 25
MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression
MILO is a compression framework that reduces the key-value cache memory used in many-shot in-context learning by applying block-wise low-rank compression. It dynamically allocates rank budgets to blocks based on information entropy, preserving important information while aggressively compressing redundant parts. Experiments on Qwen2.5 models show up to a 50% reduction in KV cache memory and a 1.8× throughput improvement with negligible performance loss on classification and reasoning tasks.