Power- and Fragmentation-aware Online Scheduling for GPU Datacenters

Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

The paper presents a fine‑grained monitoring study of a university GPU cluster, collecting data at 100 ms intervals using a lightweight tool built on Nvidia’s DCGM library. Analysis of a 10‑day period with 7,399 jobs shows that over 70 % of jobs use less than 20 % of GPU memory and 50 % have below 5 % SM utilization, while 80 % of jobs experience a large mismatch between requested and actual runtimes. The authors also observe highly periodic communication patterns in multi‑GPU jobs, indicating that significant performance gains could be achieved through improved scheduling and multitasking rather than hardware upgrades.

Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

GPU clusters have become essential computing resources for AI and scientific research in academia and industry. However, their high cost and growing demand often result in long queueing delays. While...

LLM-Powered Predictive Decision-Making for Sustainable Data Center Operations

The paper presents an LLM-based predictive scheduling system that forecasts execution time and energy consumption from source code, aiming to improve data center sustainability. By integrating these predictions into a real-time GPU allocation algorithm, the system reduces both energy use and queuing delays. In a collaboration with a data center, the approach achieved a 32% drop in energy consumption and a 30% reduction in waiting time.

A principled approach for energy-efficient training via phase-aware GPU frequency tuning

arXiv:2609.24205v1 Announce Type: cross Abstract: Modern AI model training imposes unprecedented computational demands, making it a key contributor to datacenter energy consumption. Yet a significant...

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.

Impactful scheduling for GPU clusters

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination.

ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving

arXiv:2606. 00735v1 Announce Type: cross Abstract: In distributed Mixture-of-Experts (MoE) inference, input-dependent token routing interacts with GPU performance variability to create persistent stragglers under synchronized execution, where the slowest GPU determines layer latency.

Characterizing Job Power Elasticity for Power-Flexible AI Training

The paper introduces the Power Flexibility Index (PFI) to measure how large language model (LLM) training performance changes when GPU power is reduced. Using 131 training runs on H200 and H100 GPUs, the study finds that LLM jobs have significant but variable power elasticity and identifies telemetry signals that can predict PFI during runtime. The authors demonstrate that allocating power based on PFI maximizes overall token throughput, recovering about 1.5k tokens/s per job under a 30% power reduction, which represents 63% of the gap between equal-weight and perfect-information allocations.

WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

arXiv:2607. 02391v1 Announce Type: cross Abstract: Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption.

DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference

arXiv:2606. 02982v1 Announce Type: cross Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling.

COMPASS-ABS: Reducing Fragmentation in Shared GPU Clusters for Deep Learning Training Workloads

The paper introduces COMPASS-ABS, a scheduling framework for shared GPU clusters that reduces resource fragmentation for deep learning training jobs. It defines a new metric, Scheduler-Induced Fragmentation (SIF), which does not rely on historical workload data, and presents the COMPASS algorithm that confines cluster states within an Anchor-Based Space (ABS) to keep fragmentation low. Experiments on both a physical and a simulated cluster show that COMPASS-ABS improves resource utilization and shortens job completion times by mitigating fragmentation.