arXiv Machine Learning
Characterizing High Bandwidth Flash for LLM Serving
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 30
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
arXiv:2607. 26491v1 Announce Type: cross Abstract: The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs.
arXiv Machine Learning
Jun 5
Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving
arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.
arXiv AI
Sep 16
Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions
arXiv:2609.16215v1 Announce Type: new Abstract: GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate s...
arXiv AI
Aug 10
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.
arXiv Machine Learning
Sep 15
BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference
arXiv:2609.13592v1 Announce Type: cross Abstract: GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-...
arXiv AI
Sep 16
Comparative Characterization of KV Cache Management Strategies for LLM Inference
arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...