arXiv Machine Learning

Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generation

The paper introduces PASK (Parser-Aware Structural KV Persistence), a method that leverages parser transitions to inform key‑value (KV) persistence decisions in structured generation tasks. By aligning KV compression with task‑level structured risk, PASK sets protection floors based on error sensitivity and allocates remaining KV capacity using attention‑output distortion, producing a lightweight, structure‑conditioned lookup policy. In experiments on Qwen3‑4B, PASK achieves a 17.39‑point accuracy gain over the best compressed baseline, while delivering up to 2.2× higher throughput, 3.3× lower TPOT, and 0.53× the peak GPU memory of full KV.

arXiv AI

Jun 17

AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.

By Ning Ni, Yingjie Lao

llms efficiency safety

arXiv AI

Aug 25

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...

By Srihari Unnikrishnan

llms efficiency

arXiv AI

Jul 10

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

arXiv:2607. 08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly.

By Jiantong Jiang, Peiyu Yang, Rui Zhang, Feng Liu

llms efficiency

arXiv AI

Aug 18

KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving

arXiv:2608. 15797v1 Announce Type: new Abstract: KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history.

By Minsoo Cheong, Woosang Lim, Vincent-Daniel Yun, Sungjoo Yoo

llms benchmarks

arXiv AI

Jun 6

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.

By Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu

llms fine-tuning efficiency

arXiv Machine Learning

Aug 4

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.

By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan

llms agents efficiency

arXiv AI

Jul 22

CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

arXiv:2602. 08686v3 Announce Type: replace-cross Abstract: Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction.

By Ning Yang, Chengzhi Wang, Yibo Liu, Baoliang Tian, Haijun Zhang

benchmarks

arXiv Machine Learning

Sep 10

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...

By Michael Wang, Keith Li, Roozbeh Bostandoost

llms efficiency

arXiv AI

Jul 7

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

arXiv:2607. 05147v1 Announce Type: new Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification.

By Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang

llms efficiency benchmarks

arXiv Machine Learning

3d ago

H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

arXiv:2609.24197v1 Announce Type: new Abstract: Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification b...

By Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn, Reed Meyerson, Zhenting Qi, Tianyu Wu, Eldar Kurtic, Minlan Yu, Alexandre Marques

llms diffusion efficiency

arXiv AI

Jun 9

End-to-End Context Compression at Scale

arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.

By Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov

llms rag agents efficiency

arXiv AI

Jul 23

HijackKV: New Threat in Position-Independent KV Cache Reuse

arXiv:2607. 19957v1 Announce Type: cross Abstract: Key-Value (KV) cache reduces inference latency in large language models (LLMs).

By Yichi Zhang, Zhiqi Wang, Huan Zhang, Yuchen Yang

llms efficiency

Related stories