LatentPress compresses conversational histories and long documents into continuous memory tokens that a frozen decoder can read directly, eliminating the need for text reconstruction at inference. The method achieves 4–16× compression with only a small adapter (0.1% of the decoder’s parameters) and outperforms text summaries and OCR-based compression on LongMemEval and LongBench-QA benchmarks. Writing and reading are significantly faster than traditional text summarization or OCR reconstruction, demonstrating a practical machine-facing context interface beyond text and vision.
Compressed context is usually carried as human-readable text or as rendered images that must be decoded, even when its consumer is a language model. We introduce LatentPress, which writes conversation...
The paper introduces LOHA, a context layout that compresses older tool observations into soft tokens while keeping the agent’s own turns and the last K observations in plain text, and ACD, a training method that distills full‑text predictions into this latent representation while anchoring behavior on plain text. This approach reduces context per call by up to 57% without significant loss in resolve rates, and improves instance throughput in single‑GPU serving. Experiments on SWE‑bench Verified show that K=3 yields a 43–57% compression with only modest performance impact, while larger windows favor task performance over compression.
arXiv:2609.36651v1 Announce Type: cross Abstract: Long-context reasoning in large language models incurs substantial computation and memory costs. Visual text compression (VTC) reduces input length b...
External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence. However, existing memory paradigms represent each memory item in raw text and image forms, so retrieval-based systems must pass the retrieved text or images to the generation LLMs/VLMs, resulting in high token consumption and storage pressure, making it unaffordable for resource-constrained applications.
arXiv:2606. 10572v1 Announce Type: new Abstract: External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence.
arXiv:2608.00311v2 Announce Type: replace Abstract: Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and th...
arXiv:2511. 04919v3 Announce Type: replace Abstract: Processing long documents with large language models (LLMs) is expensive: a single query over a 100K-token document can cost from tens of cents to over a dollar in API fees, depending on the model, and memory grows linearly with context length.
arXiv:2608.29897v1 Announce Type: new Abstract: Long-horizon agents need a context manager to compress growing interaction histories into a bounded working context, via passive strategies or active s...
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
arXiv:2609.25537v1 Announce Type: new Abstract: Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing laten...
arXiv:2609.36435v1 Announce Type: new Abstract: An assistant that serves the same user over a long horizon has to answer from what that user has revealed: which preferences still hold, which were rev...
The quadratic growth of attention computation and key-value (KV) cache with respect to sequence length is a central bottleneck for ultra-long-context language models and high-resolution generative mod...