arXiv Machine Learning
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 21
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
arXiv:2607. 17181v1 Announce Type: cross Abstract: Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently.
arXiv AI
Jun 24
CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.
arXiv Machine Learning
Sep 15
Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...
arXiv Machine Learning
Sep 1
WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
WiSP (Working‑Set Paging) is a routing‑aware expert pager that allows Mixture‑of‑Experts models to run on GPUs that cannot hold the entire expert pool by paging experts in and out of VRAM while preserving byte‑identical outputs. On a 24 GiB RTX 3090, WiSP doubles decode throughput compared to static offload when the model does not fit, and its companion policy MV‑WSA allocates VRAM between resident experts and KV cache based on marginal latency benefit, reducing end‑to‑end time by up to 1.19× without altering model outputs.
arXiv Machine Learning
Jun 16
Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
arXiv:2606. 06302v2 Announce Type: replace Abstract: Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput.
arXiv AI
Jun 9
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
arXiv:2606. 09613v1 Announce Type: cross Abstract: Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution.