← Back to all news
arXiv Machine Learning By Maharani A. P. Irawan, Kuntai Du, Jiayi Yao, Dongjoo Seo, Yuyang Huang, Haryadi S. Gunawi, Junchen Jiang

Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

Read the original on arXiv Machine Learning →

Gandalf is a flash‑based KV cache offloading system designed to extend SSD lifetime in large language model serving. It prioritizes full‑attention KV cache entries for SSD admission, using attention type and historical access patterns to reduce write traffic without lowering cache hit rates. Experiments show a 7.3× increase in SSD lifespan with negligible impact on hit rate across diverse workloads and hardware configurations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 2

Characterizing High Bandwidth Flash for LLM Serving

arXiv:2609.39131v1 Announce Type: new Abstract: Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer,...

By Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami
llmsagentsefficiency
More like this →
arXiv AI
Sep 24

Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

The paper introduces LM‑CXD, a CXL‑SSD design tailored for large language model (LLM) prefix caching. By aligning KV chunk management between the serving engine and the storage device, exposing NAND-to‑DRAM progress, and using device DRAM as a GPU‑accessible buffer, LM‑CXD reduces time‑to‑first‑token (TTFT) by up to 4× compared to a stock CXL‑SSD and brings performance within 1.5× of local DRAM across five LLM models. The approach also incorporates windowed prefetching and layer‑wise KV movement to hide NAND latency under limited device DRAM.

By Hyunsun Chung, Taewan Noh, Minji Kim, Joo-Young Hwang, Hong-Yeon Kim, Youngjae Kim
llmsefficiency
More like this →
arXiv Machine Learning
Oct 22

To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems

The paper introduces KVLearn, a learning‑based framework for managing key‑value (KV) cache retention in disaggregated large language model (LLM) serving systems. KVLearn comprises a Prefix Reuse Predictor that estimates reuse probability from prefix features, a Cost‑Aware Retention Score that translates this probability into keep/admit decisions based on recomputation, transfer, and storage costs, and an Adaptive Threshold Controller that tunes admission thresholds online. Integrated into a globally disaggregated serving topology, KVLearn achieves up to 56% faster time‑to‑first‑token and up to 53% reduction in inter‑node KV transfer compared to baseline caching strategies.

By Dong Liu, Yanxuan Yu, Eric Jiang, Shu Wang, Ying Nian Wu
llmsefficiencymultimodal
More like this →
arXiv AI
Jul 1

KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

arXiv:2605. 09735v2 Announce Type: replace-cross Abstract: Static-graph LLM decoders provide predictable launches, fixed tensor shapes, and low submission overhead, but online decoding exposes highly irregular KV-cache behavior: request lengths differ, EOS events arrive asynchronously, and logical histories fragment over time.

By Zhiqing Zhong, Zhijing Ye, Jian Zhang, Weijian Zheng, Bolun Sun, Xiaodong Yu
llmsefficiency
More like this →