arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.
By Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang
GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.
By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
arXiv:2609.14507v1 Announce Type: cross
Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...
By Enda Yu, Dezun Dong, Xiangke Liao
arXiv:2606. 01502v1 Announce Type: cross Abstract: Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk.
By Bole Ma, Jan Eitzinger, Harald K\"ostler, Gerhard Wellein
arXiv:2607. 04206v1 Announce Type: cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation.
By Nitin Kedia, Saurabh Agarwal, Myungjin Lee, Aditya Akella
Masked diffusion language models (dLLMs) promise faster text generation by denoising multiple tokens simultaneously, yet their real‑world serving behavior has been largely unexamined. Using LLaDA‑8B‑Instruct on a single NVIDIA H200 GPU, the study finds that request difficulty is discretized into 11 step‑count levels, short‑budget benchmarks underestimate serving variance, and only 24% of single‑request time is GPU computation, with batching mainly reducing CPU dispatch overhead. The authors also demonstrate that output quality remains stable across batch sizes and propose a batch‑timeout rule for synchronized batching under Poisson arrivals.
By Farhana Amin, Sabiha Afroz, Mona Moghadampanah, Dimitrios S. Nikolopoulos