arXiv AI October 9, 2026 By Hongyao Liu, Liuqun Zhai, Junyi Wang, Zhengru Fang, Jingshu Chen, Jun Huang

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

SparKV is an adaptive framework for on-device large language model inference that intelligently decides whether to stream or compute each key-value (KV) cache chunk, overlapping communication and computation to cut latency. It dynamically adjusts offline-generated schedules at runtime to cope with changing wireless connectivity and edge resource availability. Experiments show SparKV can reduce Time-to-First-Token by 1.3×–5.1× and lower per-request energy consumption by 1.5×–3.3× with minimal impact on response quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Oct 7

DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching

arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models...

By Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager

llms efficiency

arXiv Machine Learning

Sep 4

LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

LeanStream introduces a speculate‑and‑refine streaming framework that enables efficient on‑device inference of large language models by progressively refining computation, loading, and cache‑retention priorities using partial GPU results. This approach allows fine‑grained overlap between GPU execution and storage I/O, avoiding the trade‑offs of existing systems that serialize execution or incur redundant weight fetches. Implemented on mobile and embedded platforms, LeanStream reduces memory usage by 4.8× to 7.5× compared to prior work while improving token generation throughput by 1.6× to 2.1×.

By Renyuan Liu (Richard), Yuyang Leng (Richard), Kaiyan Liu (Richard), Yuzhou Zhong (Richard), Shaohan Hu (Richard), Chun-Fu (Richard), Chen, Peijun Zhao, Heechul Yun, Shuochao Yao

llms efficiency safety

arXiv AI

Sep 10

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

arXiv:2609.08307v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires bala...

By Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos

llms efficiency benchmarks

arXiv AI

Sep 16

Comparative Characterization of KV Cache Management Strategies for LLM Inference

arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...

By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu

llms efficiency benchmarks

arXiv AI

Jul 16

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

arXiv:2607. 13093v1 Announce Type: cross Abstract: On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy.

By Yi Li, Chen Li, Jiexiong Liu

llms rag efficiency safety

arXiv AI

Jun 3

E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

arXiv:2606. 03770v1 Announce Type: cross Abstract: Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging.

By Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La, Frank Eliassen, Phuong Hoai Ha, Peiyuan Guan

llms

One email a morning, machine-written

Related stories

DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching

LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

Comparative Characterization of KV Cache Management Strategies for LLM Inference

Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments