← Back to all news
arXiv AI By Juan Parra-Ullauri, Anthony Chiu, Hari Madhukumar, Zhaozhou Wu, Anderson Bravalheri, Alex Healing, Andy Corston-Petrie, Dimitra Simeonidou, Nektaria Efthymiou

InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network

Read the original on arXiv AI →

InferSlice is a reference architecture that enables Service-Level Objective (SLO)-driven large language model (LLM) inference across geographically distributed telco infrastructure. It integrates network slicing, GPU partitioning, elastic serving, and observability to address sovereignty, compliance, latency, and resource‑locality constraints that limit centralized cloud deployments. The architecture was evaluated on the JOINER UK national‑scale wide‑area network, demonstrating how different parallelism strategies and resource placements affect latency‑throughput regimes and scale‑out times.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 15

STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming

arXiv:2606. 13968v1 Announce Type: cross Abstract: Researchers and practitioners working with large language models face a fragmented landscape: local models are free and private but hardware limits the model size and context windows a researcher can use; institutional HPC centers offer powerful GPU resources at no marginal cost and keep data within institutional boundaries, but operate behind firewalls and are designed for batch jobs rather than interactive use; commercial cloud APIs provide frontier-model quality on demand but impose significant cost and data retention policies unsuitable for sensitive research data.

By Anas Nassar, Steve Mohr, Leonard Apanasevich, Himanshu Sharma
llmsnlpbenchmarks
More like this →
arXiv Machine Learning
Oct 15

Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

The paper investigates how different parallelism strategies—tensor parallelism (TP), pipeline parallelism (PP), and hybrids—affect the trade‑off between time‑to‑first‑token (TTFT) and inter‑token latency (ITL) in large language model inference. It introduces a framework that maps feasible TTFT‑ITL regions and Pareto frontiers for various workloads, revealing a shift from TP‑dominant to PP‑dominant configurations as prefill work increases. The authors also present an SLO‑aware selection method based on goodput, showing that stricter TTFT goals favor PP, while relaxed constraints move toward TP, and they demonstrate that longer contexts further favor PP for scalability and SLO attainment.

By Javad Mirzaei, Jeebak Mitra
llmsefficiency
More like this →