MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

MAPS is a Memory-Aware Predictive Scheduling framework designed for disaggregated large language model (LLM) serving. It uses device-assisted speculative output length prediction and uncertainty-aware calibration to establish safe output-length upper bounds, which inform a hierarchical global-local scheduling strategy that reduces queue buildup and head-of-line blocking. Experiments on real-world workloads and two LLMs demonstrate that MAPS lowers average end-to-end latency by 42.6% and tail latency by up to 84.8% compared to three state-of-the-art systems.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• efficiency
• benchmarks