arXiv AI
Jul 3
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines
arXiv:2607. 02630v1 Announce Type: cross Abstract: Hardware accelerators now sit on the critical path of online serving.
Related stories
arXiv Machine Learning
Jun 2
Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap
arXiv:2512. 10236v2 Announce Type: replace-cross Abstract: Modern ML workloads demand distributing training and inference across multiple GPUs.
Hugging Face Trending Papers
Aug 6
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
We report operational experience full-fine-tuning a 32. 76B-parameter dense model (Qwen3-32B) on 16 x NVIDIA B300 (two nodes, FSDP / ZeRO-3) -- among the first published field accounts on this accelerator.
arXiv Machine Learning
Aug 7
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
arXiv:2608. 05944v1 Announce Type: cross Abstract: We report operational experience full-fine-tuning a 32.
arXiv AI
6d ago
Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
arXiv:2608. 12123v1 Announce Type: cross Abstract: LLM-agent services repeatedly execute small deterministic transitions between model and tool calls: route an outcome, update state, and emit the next effect.
arXiv AI
Jul 7
Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes
arXiv:2607. 04668v1 Announce Type: cross Abstract: On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously.
arXiv AI
5d ago
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
arXiv:2608. 13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU.
arXiv AI
Jul 21
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
arXiv:2607. 17181v1 Announce Type: cross Abstract: Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently.
arXiv Machine Learning
Jun 8
Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction
arXiv:2209. 00188v4 Announce Type: replace-cross Abstract: Long-latency load requests continue to limit the performance of high-performance processors.
arXiv Machine Learning
Aug 10
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.
arXiv AI
Jul 28
X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference
arXiv:2607. 23264v1 Announce Type: cross Abstract: Fine-grained, device-initiated communication lets persistent GPU kernels in distributed diffusion transformer (DiT) inference issue remote stores and overlap data movement with Tensor Core computation.
arXiv Machine Learning
Jul 14
Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
arXiv:2607. 11368v1 Announce Type: cross Abstract: Reported serving speedups from quantized kernels typically bundle the weight format, the kernel, and the inference runtime into one number.