Optimum-NVIDIA Unlocking blazingly fast LLM inference in just 1 line of code

Dissecting GPU Utilization for LLM Inference on Nvidia Hopper

The paper investigates how a single SM utilization metric can misrepresent the true workload of large language model (LLM) inference on Nvidia Hopper GPUs. By profiling vLLM with FlashAttention‑3 and cuBLASLt on an H100 NVL across various phases (cold prefill, warm prefill, and decode) and varying sequence length and batch size, the authors replace the single utilization figure with eight detailed counter‑validated views. These views, tied to specific Nsight Compute counters or formulas, reveal how factors such as fragment fill, occupancy limits, stall signatures, wave quantization, and kernel selection create utilization gaps across four production models and six per‑layer kernel roles.

🚀 Accelerating LLM Inference with TGI on Intel Gaudi

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.

RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems

arXiv:2609.12551v2 Announce Type: replace-cross Abstract: AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based....

Tile-Level Activation Overlap for Efficient LLM Inference

arXiv:2607. 02521v1 Announce Type: cross Abstract: SwiGLU is the dominant MLP activation in modern large language models, yet its intermediate tensor materialization costs 9-37% of MLP execution time.

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.

KLineage: Recovering the Missing When of Kernel Optimization by Deoptimizing Experts

arXiv:2605.28213v2 Announce Type: replace Abstract: LLM-based agents are increasingly used to generate GPU kernels, but they often struggle to determine when an optimization is sound because its requ...

KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?

arXiv:2607. 16241v1 Announce Type: cross Abstract: Recent large language models (LLMs) can generate custom CUDA kernels that appear to outperform PyTorch on benchmarks such as KernelBench.

Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B

arXiv:2607. 04244v1 Announce Type: new Abstract: This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.

Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

arXiv:2607. 19353v1 Announce Type: new Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets.

Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference

Celty introduces a co-designed sparse format, GPU kernel, and SIMT microarchitecture to efficiently handle Sparse Matrix‑Sparse Vector (SpMSpV) workloads in large language model inference. Its Run‑Length Compressed CSC (RLC‑CSC) format allows vectorized loading of compressed weight columns and skips memory accesses by exploiting both weight pruning and activation sparsity. The Celty Sparse SIMT Core adds a pipelined RLC decoder that eliminates software index reconstruction and uses local registers for conflict‑free accumulation, achieving up to 5.3× speedup over cuBLAS at 70% dual‑sparsity.