arXiv Machine Learning September 7, 2026 By Di Chai, Leye Wang, Zeshen Su, Zhiguo Xia, Zhihang Yu

KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU

Read the original on arXiv Machine Learning →

KVMem is a KV-context virtualization system that allows large language model agents to maintain workspaces exceeding both GPU key‑value capacity and the model’s native context window. It stores overflowed history as paged KV state across GPU memory, host memory, and NVMe, using lightweight, model‑native attention‑space indexes to retrieve relevant historical blocks. Evaluations on long‑context agent benchmarks show that KVMem improves task utility and inference efficiency, enabling up to one million‑token workspaces on consumer GPUs and achieving interactive responsiveness in local deployments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 30

InferScale: GPU-Native KV Injection for Personalized LLM Serving

arXiv:2607. 27090v1 Announce Type: cross Abstract: Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests.

By Peter Li, Prashant Pandey

llms rag fine-tuning efficiency

Hugging Face Trending Papers

Jul 29

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.

llms rag fine-tuning efficiency

arXiv AI

Aug 25

Context as an Environment: Programmatic Context Management for Long-Horizon Agents

arXiv:2608.21690v1 Announce Type: new Abstract: LLM agents increasingly take on long-running tasks whose history grows far beyond a single model context window. Existing approaches compress earlier i...

By Yin Lin, Elaine Ang, Erkang Zhu, Bolin Ding, Jingren Zhou

llms agents

arXiv AI

Jun 6

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.

By Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu

llms fine-tuning efficiency

arXiv Machine Learning

Jun 3

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.

By Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao, Bin Cui

llms rag agents efficiency

Hugging Face Trending Papers

Aug 21

Context as an Environment: Programmatic Context Management for Long-Horizon Agents

LLM agents increasingly take on long-running tasks whose history grows far beyond a single model context window. Existing approaches compress earlier interactions or extract selected information into...

llms agents

One email a morning, machine-written

Related stories

InferScale: GPU-Native KV Injection for Personalized LLM Serving

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Context as an Environment: Programmatic Context Management for Long-Horizon Agents

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

Context as an Environment: Programmatic Context Management for Long-Horizon Agents