arXiv Machine Learning
mzCache: On-Device LLM Memory Management under Multitasking
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 4
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
LeanStream introduces a speculate‑and‑refine streaming framework that enables efficient on‑device inference of large language models by progressively refining computation, loading, and cache‑retention priorities using partial GPU results. This approach allows fine‑grained overlap between GPU execution and storage I/O, avoiding the trade‑offs of existing systems that serialize execution or incur redundant weight fetches. Implemented on mobile and embedded platforms, LeanStream reduces memory usage by 4.8× to 7.5× compared to prior work while improving token generation throughput by 1.6× to 2.1×.
arXiv Machine Learning
Sep 21
TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
arXiv:2609.21172v1 Announce Type: new Abstract: Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications...
arXiv Machine Learning
Jun 15
Efficient On-Device Diffusion LLM Inference with Mobile NPU
arXiv:2606. 13740v1 Announce Type: new Abstract: Diffusion large language models (dLLMs) accelerate generation by denoising multiple tokens in parallel, making them attractive for latency-sensitive mobile inference.
arXiv AI
21h ago
FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
arXiv:2611.07542v1 Announce Type: new Abstract: In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference...
Hugging Face Trending Papers
1d ago
FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidth memory (HBM) capac...
arXiv Machine Learning
Jul 30
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
arXiv:2607. 26491v1 Announce Type: cross Abstract: The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs.