arXiv:2606. 04557v1 Announce Type: cross Abstract: Large Language Models can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries.
Latent Compaction is a test‑time training method that learns a compact, document‑specific key–value cache for large‑context language models. It initializes cache entries using reconstruction importance, teacher attention, and key‑space diversity, then fits attention outputs and log‑normalizers to the full cache before jointly optimizing the entire cache via end‑to‑end distillation. Experiments on RULER, LongBench, and Phonebook Lookup show significant accuracy gains over budget‑matched compression baselines, achieving 96.7% accuracy at 32× compression on RULER and transferring well across model scales and families.
arXiv:2511. 04919v3 Announce Type: replace Abstract: Processing long documents with large language models (LLMs) is expensive: a single query over a 100K-token document can cost from tens of cents to over a dollar in API fees, depending on the model, and memory grows linearly with context length.
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
Recent work in machine learning has demonstrated that transformer-based language model agents can communicate state not only through text, but also directly in latent space via key-value (KV) caches....
The paper introduces QCOC, a method for compressing the key‑value cache used in tabular in‑context learning. QCOC clusters in‑context example states into joint‑KV prototypes, calibrates them against query vectors, and preserves multiplicities to maintain accuracy while achieving high compression. Experiments on 64 OpenML‑CC18 datasets and seven long tables show QCOC outperforms other compression and retrieval methods, achieving up to a 10.5× compression ratio and 508× speed‑up over dynamic retrieval baselines while staying within 0.23 percentage points of full‑context accuracy.
CacheGraft is a retrieval‑augmented generation system that enriches a small language model’s latent space by injecting fused key‑value cache representations of corpus passages, encoded offline by a larger model. The approach eliminates the need to run the larger model at inference time, and on five QA benchmarks it improves answer quality in 29 of 30 comparisons, with 17 statistically significant gains. These results demonstrate that latent‑space document enrichment can boost RAG performance, especially when only a small model can be deployed.
arXiv:2609.39329v1 Announce Type: new Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
arXiv:2607. 05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks.
arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...
LowRAM introduces a two‑stage compression scheme for the KV cache used in autoregressive decoding. It first amortises values by predicting them from keys with a lightweight per‑head predictor, storing only residuals for poorly reconstructed tokens, achieving roughly 4× near‑lossless value compression. Then it applies cross‑layer low‑rank key compression, resulting in an asymmetric pipeline that can boost peak decode throughput by up to 5× while maintaining near‑lossless performance, and can be further combined with quantisation and eviction for extreme compression ratios.
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...