arXiv AI
Jul 24
Error Certificates for KV-Cache Eviction via Randomized Design
arXiv:2607. 21475v1 Announce Type: cross Abstract: Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest.
Error Certificates for KV-Cache Eviction via Randomized Design
Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest. We prove that this design cannot know what it destroyed: evicted values can be altered so that everything the serving system retains is unchanged while the true attention-output error grows arbitrarily, so no serving-time estimator of that error is consistent.
Related stories
arXiv AI
Aug 7
Runtime Observability for Heterogeneous Attention Memory
arXiv:2608. 05863v1 Announce Type: new Abstract: Modern models no longer keep a plain KV cache: latent caches, learned sparse selectors and recurrent states each carry the model's memory in a different form, and each fails differently under compression.
arXiv Machine Learning
Aug 11
CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents
arXiv:2608. 07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations.
arXiv Machine Learning
Jun 24
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
arXiv:2606. 23961v1 Announce Type: new Abstract: Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inference stream.
arXiv AI
Aug 11
Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference
arXiv:2608. 09225v1 Announce Type: cross Abstract: The key-value (KV) cache is the primary throughput optimization in modern large language model (LLM) inference, enabling prefix reuse across requests.
arXiv AI
Jun 30
Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds
arXiv:2606. 29654v1 Announce Type: new Abstract: Multi-agent deliberation among LLMs can improve reasoning, but deployment requires deciding when the current answer is reliable enough to act on and when it should be escalated to human review.
arXiv AI
Aug 11
Private Anytime Selective-Risk Certification for Federated Retrieval-Augmented Generation: Guarantees and Empirical Limits
arXiv:2608. 07913v1 Announce Type: cross Abstract: Selective-risk certificates promise that accepted outputs meet a declared error target.
arXiv AI
Aug 3
WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
arXiv:2607. 28699v1 Announce Type: cross Abstract: KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now.
arXiv AI
Jun 17
AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor
arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.
arXiv Machine Learning
Aug 4
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
arXiv:2608. 01247v1 Announce Type: cross Abstract: Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets.
Hugging Face Trending Papers
Aug 8
Private Anytime Selective-Risk Certification for Federated Retrieval-Augmented Generation: Guarantees and Empirical Limits
Selective-risk certificates promise that accepted outputs meet a declared error target. We develop Fed-SRC, a score-agnostic certificate for federated, differentially private, adaptively monitored retrieval-augmented generation.
arXiv AI
Jul 21
High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
arXiv:2607. 16248v1 Announce Type: cross Abstract: Long-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads.