arXiv Machine Learning June 25, 2026 By Yipin Guo, Siddharth Joshi

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

Read the original on arXiv Machine Learning →

arXiv:2605. 01708v3 Announce Type: replace-cross Abstract: Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 2

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

arXiv:2607. 00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost.

By Sheng Qiang, Ruiwei Chen, Yinpeng Wu, Jinyu Gu, Zhichao Hua, Yubin Xia, Binyu Zang, Haibo Chen

llms efficiency

arXiv Machine Learning

Aug 11

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.

By Anthony. Lui, Mohamed. Elsaied, N. P. Savani

llms efficiency

arXiv AI

Jun 24

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.

By Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang

llms benchmarks

arXiv AI

Jun 16

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.

By Chao Fei, Panos Kalnis

llms efficiency

arXiv Machine Learning

Jun 16

Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving

arXiv:2606. 06302v2 Announce Type: replace Abstract: Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput.

By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi

llms efficiency

arXiv Machine Learning

Jun 5

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.

By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi

llms efficiency

Related stories

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving