arXiv:2606. 04557v1 Announce Type: cross Abstract: Large Language Models can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries.
arXiv:2511. 04919v3 Announce Type: replace Abstract: Processing long documents with large language models (LLMs) is expensive: a single query over a 100K-token document can cost from tens of cents to over a dollar in API fees, depending on the model, and memory grows linearly with context length.
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
arXiv:2607. 05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks.
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
arXiv:2609.22100v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) improves language models with retrieved evidence, but processing many long passages is costly and can introduce...
arXiv:2606. 15734v1 Announce Type: cross Abstract: Continual post-training enables models to absorb emerging knowledge after deployment, but repeatedly updating shared parameters can accumulate weight drift, potentially causing catastrophic forgetting and degrading general capabilities.
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
The paper introduces DEPT, a method that trains a single decoder-only large language model to both expand queries and encode documents for retrieval. By preserving document embeddings close to their initial cached values while allowing gradients to flow through the generator, DEPT stabilizes retrieval targets and enables efficient index reuse and online hard‑negative mining. Experiments on the BEIR benchmark with Qwen3‑4B‑Instruct‑2507 and LLaMA‑3.2‑3B‑Instruct show that DEPT outperforms training‑free, independently trained, and staged unified baselines, with ablations confirming the benefits of preservation, whitening, end‑to‑end expansion training, and online negatives.