arXiv Machine Learning October 15, 2026 By Javad Mirzaei, Jeebak Mitra

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Read the original on arXiv Machine Learning →

The paper investigates how different parallelism strategies—tensor parallelism (TP), pipeline parallelism (PP), and hybrids—affect the trade‑off between time‑to‑first‑token (TTFT) and inter‑token latency (ITL) in large language model inference. It introduces a framework that maps feasible TTFT‑ITL regions and Pareto frontiers for various workloads, revealing a shift from TP‑dominant to PP‑dominant configurations as prefill work increases. The authors also present an SLO‑aware selection method based on goodput, showing that stricter TTFT goals favor PP, while relaxed constraints move toward TP, and they demonstrate that longer contexts further favor PP for scalability and SLO attainment.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 8

Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds

arXiv:2604. 07472v2 Announce Type: replace Abstract: Serving large language model (LLM) inference in cloud environments requires jointly optimizing model selection, GPU provisioning, parallelism configuration, and workload routing under latency, accuracy, memory, and budget constraints.

By Jiaming Cheng, Duong Tung Nguyen

llms

arXiv AI

Jul 7

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.

By Ferran Agullo, Joan Oliveras, Chen Wang, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Ll. Berral

llms efficiency benchmarks

arXiv Machine Learning

Aug 18

Multi-Bin Batching for Increasing LLM Inference Throughput

arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.

By Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani

llms efficiency

arXiv Machine Learning

Jun 9

Efficient Scaling of LLM Training with Flexible Context Parallelism

arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).

By Yifan Niu, Han Xiao, Dongyi Liu, Wei Zhou, Jia Li

llms efficiency

arXiv AI

Jun 10

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.

By Wenxin Wang, Yule Hou, Yu Ji, Peng Qu, Youhui Zhang

efficiency

arXiv AI

Jun 16

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints

arXiv:2504. 11320v4 Announce Type: replace-cross Abstract: Large language models now serve millions of users daily, with providers incurring costs exceeding $700,000 per day.

By Ruicheng Ao, Gan Luo, David Simchi-Levi, Xinshang Wang

llms benchmarks safety

One email a morning, machine-written

Related stories

Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

Multi-Bin Batching for Increasing LLM Inference Throughput

Efficient Scaling of LLM Training with Flexible Context Parallelism

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints