arXiv Machine Learning October 28, 2026 By YuXuan Lin, KeQiang Wu, Kingsum Chow

DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

Read the original on arXiv Machine Learning →

DualPace is an LLM serving runtime that decouples scheduling between a GPU engine and an AMX‑enabled CPU engine, allowing the CPU to independently batch and advance decodes while still cooperating with the GPU for dense stages. This design expands key‑value cache capacity into host memory and reduces bursty CPU progress, leading to significant throughput gains—up to 43.2 % over NEO and 90.6 % over vLLM—without increasing latency. The system also maintains low time‑to‑first‑token and supports higher request rates at comparable tail latency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 3

Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.

By Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, Victor R\"uhle

llms benchmarks

arXiv Machine Learning

Jun 2

DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

arXiv:2511. 04791v2 Announce Type: replace Abstract: Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases.

By Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark Hill, Murali Annavaram

llms efficiency benchmarks

arXiv Machine Learning

Jul 28

FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon

arXiv:2607. 22785v1 Announce Type: cross Abstract: Apple-Silicon SoCs share CPU, GPU, and Neural Engine over one unified memory system, raising the question of whether transformer inference can be accelerated by splitting single operators across units.

By Om Mohite

llms efficiency

arXiv Machine Learning

Sep 15

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...

By Enda Yu, Dezun Dong, Xiangke Liao

arXiv Machine Learning

Aug 21

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.

By Aman Sunesh, Ali Alshehhi, Hivansh Dhakne

llms efficiency benchmarks

arXiv Machine Learning

Aug 4

HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.

By Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica, Jae W. Lee

llms diffusion efficiency benchmarks

One email a morning, machine-written

Related stories

Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon

Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval