The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
CacheGraft is a retrieval‑augmented generation system that enriches a small language model’s latent space by injecting fused key‑value cache representations of corpus passages, encoded offline by a larger model. The approach eliminates the need to run the larger model at inference time, and on five QA benchmarks it improves answer quality in 29 of 30 comparisons, with 17 statistically significant gains. These results demonstrate that latent‑space document enrichment can boost RAG performance, especially when only a small model can be deployed.
Latent Compaction is a test‑time training method that learns a compact, document‑specific key–value cache for large‑context language models. It initializes cache entries using reconstruction importance, teacher attention, and key‑space diversity, then fits attention outputs and log‑normalizers to the full cache before jointly optimizing the entire cache via end‑to‑end distillation. Experiments on RULER, LongBench, and Phonebook Lookup show significant accuracy gains over budget‑matched compression baselines, achieving 96.7% accuracy at 32× compression on RULER and transferring well across model scales and families.
arXiv:2605. 22863v2 Announce Type: replace Abstract: LLM agents today communicate via text, which incurs considerable latency and information loss due to the need to autoregressively decode the sharer model's state and encode at the receiver model.
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
The paper introduces DEPT, a method that trains a single decoder-only large language model to both expand queries and encode documents for retrieval. By preserving document embeddings close to their initial cached values while allowing gradients to flow through the generator, DEPT stabilizes retrieval targets and enables efficient index reuse and online hard‑negative mining. Experiments on the BEIR benchmark with Qwen3‑4B‑Instruct‑2507 and LLaMA‑3.2‑3B‑Instruct show that DEPT outperforms training‑free, independently trained, and staged unified baselines, with ablations confirming the benefits of preservation, whitening, end‑to‑end expansion training, and online negatives.