arXiv AI
Jun 16
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.
LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference
Related stories
arXiv AI
Oct 7
DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching
arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models...
arXiv AI
Jun 29
End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference
arXiv:2606. 27743v1 Announce Type: cross Abstract: Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph regardless of the runtime environment.
arXiv AI
3d ago
InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network
InferSlice is a reference architecture that enables Service-Level Objective (SLO)-driven large language model (LLM) inference across geographically distributed telco infrastructure. It integrates network slicing, GPU partitioning, elastic serving, and observability to address sovereignty, compliance, latency, and resource‑locality constraints that limit centralized cloud deployments. The architecture was evaluated on the JOINER UK national‑scale wide‑area network, demonstrating how different parallelism strategies and resource placements affect latency‑throughput regimes and scale‑out times.
arXiv Machine Learning
Sep 11
Optimizing AI Inference Across the Deployment Stack
The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.
arXiv AI
Jul 7
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.
arXiv AI
Aug 20
Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
The paper presents a method for distributing large language model inference across multiple Intel AI PCs by splitting the model into pipeline shards, each pre‑compiled into an OpenVINO graph. Three key techniques—beam_idx Gather to enable GPU optimizations, speculative decoding on stateful models, and interleaved micro‑batching—allow a two‑node Llama 3.1 8B INT4 pipeline to serve two users at 1.79× the throughput of a single‑node model, while a four‑node deployment can run a 70B model that no single PC can hold. The authors provide code, benchmark logs, and reproduction scripts on GitHub.
arXiv AI
Aug 10
Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.
arXiv AI
Sep 10
A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
arXiv:2609.08307v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires bala...
arXiv AI
Aug 14
SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
arXiv:2608. 13076v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands.
arXiv AI
Jun 10
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.
arXiv AI
3d ago
Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference
The paper introduces S‑SALAAD, a framework that jointly learns model weights, low‑rank, and structured sparse components during training. It supports hardware‑compatible sparsity patterns and couples structure learning with budget‑aware compression and dedicated CPU/GPU kernels, allowing efficient inference from a single checkpoint across multiple memory budgets. Experiments on Llama2 and Qwen3 demonstrate improved perplexity, memory usage, and latency on various hardware platforms compared to baselines.