SAKURA is a cloud-edge system that delivers reusable attention key‑value state to mobile devices for diffusion‑based image editing while respecting a bandwidth budget. It compresses the state with a token‑importance‑aware method, dynamically selects cache representations based on network conditions, and overlaps transfer with on‑device computation. Experiments on a Snapdragon 8 Elite Gen 5 phone show that SAKURA cuts latency by up to 3.27× and reduces cache storage by 3.06× compared to uncompressed caches, all while maintaining image quality.
arXiv:2609.21172v1 Announce Type: new Abstract: Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications...
arXiv:2606. 13740v1 Announce Type: new Abstract: Diffusion large language models (dLLMs) accelerate generation by denoising multiple tokens in parallel, making them attractive for latency-sensitive mobile inference.
arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models...
Retrieval-augmented and agentic workloads repeatedly prefill recurring predictable structured inputs (which we call "spans") such as documents and code files. Yet, prefix caching in engines such as vLLM cannot reuse their KV entries unless they share identical prefixes with another request, while Position-Independent Caching (PIC) implementations within production-grade inference servers typically either require substantial server code changes or keep KV state outside the server, incurring host-to-device transfer overhead.
arXiv:2604. 26508v2 Announce Type: replace-cross Abstract: Deploying Vision-Language Models (VLMs) on edge devices remains challenging due to their substantial computational and memory demands, which exceed the capabilities of resource-constrained embedded platforms.
arXiv:2606. 13126v1 Announce Type: cross Abstract: Retrieval-augmented and agentic workloads repeatedly prefill recurring predictable structured inputs (which we call "spans") such as documents and code files.
SparKV is an adaptive framework for on-device large language model inference that intelligently decides whether to stream or compute each key-value (KV) cache chunk, overlapping communication and computation to cut latency. It dynamically adjusts offline-generated schedules at runtime to cope with changing wireless connectivity and edge resource availability. Experiments show SparKV can reduce Time-to-First-Token by 1.3×–5.1× and lower per-request energy consumption by 1.5×–3.3× with minimal impact on response quality.
arXiv:2412. 18911v3 Announce Type: replace-cross Abstract: Diffusion Transformers (DiT) have become the dominant methods in image and video generation yet still suffer substantial computational costs.
The paper introduces GAAC, a lossless key‑value (KV) compression method for large language model inference on personal devices. GAAC encodes BF16 exponents with per‑channel base‑plus‑offset codes, adapts code widths across small groups, and reconstructs KV tiles on the fly to avoid full‑cache decompression. Experiments show 1.4× compression for BF16 KV, speedups of up to 1.27× in decode attention, and a 26% reduction in resident KV memory while boosting throughput on NVIDIA and Apple hardware.
DIFFLET is a serving engine for diffusion transformer models on AWS Trainium that allows five model pipelines to share a single compilation backend. It splits joint attention across cores by attention head and uses a host controller to decide whether to run the denoiser or reuse previous outputs at each denoising step. Across five image and video model families, DIFFLET’s fused attention accelerates denoiser steps by 1.2–4.3× compared to PyTorch/XLA, and resident serving is 1.5–3.4× faster than standalone runs that reload weights.
arXiv:2607. 13095v1 Announce Type: cross Abstract: We present a full-pipeline inference optimization for the MiMo-V2.
The paper introduces Iris, a framework that extends typed decision‑making models to multimodal settings where candidates can be images, text, or hybrids. Iris preserves key properties such as candidate‑order invariance, scalability, and cache reuse, and achieves high accuracy on a visual‑candidate benchmark while reducing latency through efficient encoding strategies. The authors also demonstrate significant speedups on GPU systems and outperform larger baseline models on MetaEncoder benchmarks.