Flash-dLLM is a training‑free inference acceleration framework that improves the speed and memory efficiency of Diffusion Large Language Models (dLLMs). It tackles GPU memory I/O bottlenecks by introducing an I/O‑aware fused KV‑cache kernel and then employs a draft‑and‑verify decoding strategy that uses the dLLM itself as both drafter and verifier. Experiments on mathematical reasoning and code‑generation tasks show Flash‑dLLM outperforms existing acceleration methods, achieving up to 11.0× speedups over the Elastic‑Cache baseline.
arXiv:2606. 26120v1 Announce Type: cross Abstract: Diffusion Large Language Models (dLLMs) offer a promising alternative to autoregressive models, excelling in text generation tasks due to their bidirectional attention mechanisms.
The paper "LLM Inference in a Flash!" proposes an integer‑only quantization scheme and a dictionary‑based KV cache compression technique to enable large language model inference on compute‑in‑flash (CIF) devices. By eliminating floating‑point operations and reducing KV cache traffic through sparse dictionary coding, the authors achieve minimal accuracy loss while cutting dynamic KV cache traffic by 15× on Llama‑3.1‑8B and Qwen‑2.5‑7B models.
FlashOffloading is a host‑DRAM‑centric inference architecture that fully offloads the key‑value (KV) cache for large language model serving, addressing the high‑bandwidth memory (HBM) capacity wall that limits concurrency. It introduces a rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index with zero miss‑lookup overhead, and a compute–load overlapped kernel to reduce KV offloading latency across the pipeline. Experiments on an Ascend910C platform with GLM‑5.2 show up to 1.38× prefill throughput and 1.91× decode throughput gains under a 30 ms time‑per‑output‑token constraint.
arXiv:2511. 21759v2 Announce Type: replace-cross Abstract: Diffusion-based large language models (dLLMs) have recently gained significant attention for their exceptional performance and inherent potential for parallel decoding.
FlashOffloading is a host-DRAM-centric inference architecture that fully offloads the key-value cache for large language model serving, addressing the high-bandwidth memory capacity wall that limits concurrency. It introduces a rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index, and a compute–load overlapped kernel to minimize offloading latency across the pipeline. Experiments with GLM‑5.2 on an Ascend910C platform show up to 1.38× higher prefill throughput and 1.91× higher decode throughput under a 30 ms time‑per‑output‑token constraint.
arXiv:2506. 06295v2 Announce Type: replace-cross Abstract: Autoregressive Models (ARMs) have long dominated the landscape of Large Language Models.
GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.
arXiv:2607. 16339v1 Announce Type: new Abstract: Diffusion-based Large Language Models(DLLMs) enable parallel generation via Semi-Autoregressive (SAR) decoding in text generation.
DynaTE is a hardware–software co‑design that accelerates diffusion‑based large language models (dLLMs) by dynamically adapting execution to token states. It skips low‑utility token computations, uses a dimension‑reconfigurable processing array, refines locally dependent tokens with FLDD, and streams vocabulary to handle irregular token patterns. These techniques yield 2.05–2.78× speedup and 2.99–3.93× higher energy efficiency compared to existing dLLM accelerators, and outperform Jetson AGX Orin by 2.55× speedup and 6.07× energy efficiency.
StrataFlux is a framework that improves the accuracy of low‑bit quantized large language model inference by selectively streaming higher‑precision data from host RAM and SSDs. It co‑designs refinement representation, selection, and delivery to reduce transfer volume, overlap data delivery with computation, and reuse fetched precision across decoding steps. Experiments on GPUs and unified‑memory platforms show that StrataFlux can lower perplexity and raise accuracy for models like Llama‑3.1‑8B‑Instruct and Qwen3‑8B while maintaining high throughput and using roughly half the GPU memory compared to conventional 4‑bit quantization.
arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.
arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...