arXiv AI
Jul 3
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
Disaggregation Is a Thousand-GPU Problem
The article discusses the conditions under which separating the prefill phase from the decode phase in large language model inference is beneficial. It explains that only when three specific criteria are met does this split pay off, and it recommends using chunked prefill as the default approach when operating below a certain GPU threshold.
Related stories
Towards Data Science
Jun 3
I Built a C++ Backend So My GPU Would Stop Eating Air
A comprehensive guide to optimizing LLM inference by eliminating padding overhead with hardware-aware sequence packing. The post I Built a C++ Backend So My GPU Would Stop Eating Air appeared first on Towards Data Science .
arXiv Machine Learning
Sep 23
Disaggregated Quantization: Specializing LLM Prefill and Decode
arXiv:2609.26333v1 Announce Type: new Abstract: Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce me...
Towards Data Science
Jun 19
GPU-Resident Top-K for Agentic RAG: I Built a CUDA Kernel So My Retrieval Step Would Stop Bouncing Off the GPU
The PCIe transfer latency is silently bottlenecking your agentic inference. Here is how building a custom device-resident vector search kernel bypasses the CPU to unlock deterministic microsecond tail latencies.
Hugging Face Trending Papers
Jun 9
Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models
Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can outperform dense attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the decode length.
arXiv Computation and Language
Sep 1
Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models
arXiv:2606.10537v2 Announce Type: replace Abstract: Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with...
arXiv Machine Learning
Sep 14
Dissecting GPU Utilization for LLM Inference on Nvidia Hopper
The paper investigates how a single SM utilization metric can misrepresent the true workload of large language model (LLM) inference on Nvidia Hopper GPUs. By profiling vLLM with FlashAttention‑3 and cuBLASLt on an H100 NVL across various phases (cold prefill, warm prefill, and decode) and varying sequence length and batch size, the authors replace the single utilization figure with eight detailed counter‑validated views. These views, tied to specific Nsight Compute counters or formulas, reveal how factors such as fragment fill, occupancy limits, stall signatures, wave quantization, and kernel selection create utilization gaps across four production models and six per‑layer kernel roles.
arXiv Machine Learning
Jun 30
Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path
arXiv:2606. 29565v1 Announce Type: new Abstract: A stateless inference server (vLLM, SGLang, TensorRT-LLM) idles between requests while the accelerator waits; a stateful session reclaims that idle time.
arXiv Machine Learning
Jul 28
FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon
arXiv:2607. 22785v1 Announce Type: cross Abstract: Apple-Silicon SoCs share CPU, GPU, and Neural Engine over one unified memory system, raising the question of whether transformer inference can be accelerated by splitting single operators across units.
arXiv AI
5d ago
Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
The paper introduces Decode‑Latency Feedback Prefill (DLFP), a model‑free controller that adjusts prefill chunk sizes during concurrent autoregressive inference to reduce interference between new and ongoing requests. Implemented in vLLM, DLFP achieves significant reductions in P99 inter‑token latency on Qwen3‑0.6B while maintaining output correctness and SLO compliance, though it fails to generalize to larger models or multi‑GPU setups. The study highlights the limits of this approach and suggests the need for a completion‑timed controller for broader applicability.
Towards Data Science
Jul 21
How Much of a Data Science Workflow Can Run on a GPU Today? Part 1: Accelerating Data Preparation
Exploring GPU acceleration with cuDF, cudf. pandas, and the Polars GPU Engine The post How Much of a Data Science Workflow Can Run on a GPU Today?
Towards Data Science
Jun 11
When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI
Why “average utilization” lies about how full your GPUs really are The post When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI appeared first on Towards Data Science .