Meganeura: Portable GPU Training and Inference through Vulkan and Metal
arXiv:2608. 01563v1 Announce Type: new Abstract: Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries.
Obaleth is a compiler that incorporates GPU-initiated communication into a kernel’s intermediate representation, enabling co‑optimization of communication and computation. It generates specialized transport code for NVIDIA and AMD GPUs, optimizing buffer lifetimes, memory ordering, synchronization, and batching. Benchmarks show significant speedups: up to 3.3× faster small‑message sends on H100 GPUs, 1.25×–2.3× faster all‑to‑all on 256 GPUs compared to NVSHMEM and NCCL GIN, and 1.29× faster Qwen3‑235B inference on 16 B200 GPUs in vLLM.
arXiv:2608. 01563v1 Announce Type: new Abstract: Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries.
arXiv:2606. 09200v1 Announce Type: cross Abstract: The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems.
arXiv:2609.36070v1 Announce Type: cross Abstract: AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single...
arXiv:2601.22585v2 Announce Type: replace-cross Abstract: The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. Howev...
The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.
DualPace is an LLM serving runtime that decouples scheduling between a GPU engine and an AMX‑enabled CPU engine, allowing the CPU to independently batch and advance decodes while still cooperating with the GPU for dense stages. This design expands key‑value cache capacity into host memory and reduces bursty CPU progress, leading to significant throughput gains—up to 43.2 % over NEO and 90.6 % over vLLM—without increasing latency. The system also maintains low time‑to‑first‑token and supports higher request rates at comparable tail latency.
arXiv:2603. 16428v2 Announce Type: replace-cross Abstract: Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs.
ThunderEP is a new communication design for expert parallelism on PCIe-connected consumer GPUs that eliminates relay hops, uses DMA engines to avoid GPU compute contention, and reduces CPU polling overhead. Integrated into vLLM, it outperforms NCCL on RTX 4090 and RTX 5090 GPUs, delivering average speedups of 2.00× for dispatch, 1.53× for combine, and up to 1.66× end‑to‑end over existing MoE inference frameworks.
The paper introduces LLVMhammer, a Rowhammer attack that induces a bit‑flip in Triton’s MLIR/LLVM compilation pipeline, causing the generation of a suboptimal GPU kernel that persists across inference runs. A single successful bit‑flip can increase LLM inference latency by 4–16 % on NVIDIA RTX 4090 and AMD RX 7900 XTX GPUs, and can also degrade output quality, lowering BERTScore to as low as 25 %. The attack is GPU‑vendor‑agnostic and demonstrates that memory corruption in the compiler can have lasting performance and security impacts on large‑language‑model inference.
The paper investigates how well common large‑language‑model (LLM) inference optimizations perform on Apple Silicon compared to server‑class NVIDIA GPUs. Four optimization families—KV‑cache pruning, speculative decoding, ported GPU kernels, and post‑training quantization—are evaluated on four Apple machines under a single‑request decode workload, with matched RTX 4090 comparisons where possible. Results show that KV‑cache pruning can boost Apple throughput up to 1.75×, while speculative decoding actually reduces throughput on Apple; CUDA kernel designs transfer only partially, and quantization gains vary by machine, indicating that speedups depend on the specific technique, runtime, hardware, and workload regime.
arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.
AMMA is a multi‑chiplet, memory‑centric architecture designed to reduce latency in large‑context attention for language models. It replaces GPU compute dies with HBM‑PNM cubes, doubling memory bandwidth and employing a logic‑die microarchitecture, two‑level hybrid parallelism, and a reordered collective flow to minimize communication overhead. Design‑space exploration and evaluations show AMMA achieves 15.5× lower attention latency and 6.9× lower energy consumption compared with the NVIDIA H100.