arXiv AI
LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 16
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.
arXiv AI
Oct 7
DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching
arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models...
arXiv AI
Jun 29
End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
arXiv:2606. 27743v1 Announce Type: cross Abstract: Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph regardless of the runtime environment.
arXiv AI
3d ago
InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network
InferSlice is a reference architecture that enables Service-Level Objective (SLO)-driven large language model (LLM) inference across geographically distributed telco infrastructure. It integrates network slicing, GPU partitioning, elastic serving, and observability to address sovereignty, compliance, latency, and resource‑locality constraints that limit centralized cloud deployments. The architecture was evaluated on the JOINER UK national‑scale wide‑area network, demonstrating how different parallelism strategies and resource placements affect latency‑throughput regimes and scale‑out times.
arXiv Machine Learning
Sep 11
Optimizing AI Inference Across the Deployment Stack
The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.
arXiv AI
Jul 7
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.