SparseDitto is an agentic sparse compilation framework that jointly synthesizes representation, execution schedule, and hardware mapping for sparse matrix computations on GPUs. It uses structural analysis, a learned template-ranking prior, and LLM-guided lowering to generate CUDA code, with target-GPU profiling refining the plan. The framework supports multiple operators such as SpMV, SpMM, and SpGEMM, adapts to different hardware, and achieves significant speedups over cuSPARSE, including up to 146.61× on certain matrices and 3.39× acceleration for full-batch GCN training.
arXiv:2608. 05033v1 Announce Type: cross Abstract: Sparse matrix kernels are fundamental to scientific computing, graph analytics, and machine learning.
arXiv:2604.18616v2 Announce Type: replace-cross Abstract: LLM coding agents can generate correct GPU kernels, but their performance still trails expert libraries. Reaching peak throughput requires co...
arXiv:2604. 01489v2 Announce Type: replace Abstract: High-performance GPU kernels are critical to modern machine learning systems, yet developing them remains a manual, expert-driven process.
The paper introduces a hardware-software co‑design framework that compresses Mixture‑of‑Experts (MoE) model weights into low‑precision, hardware‑native sparse representations, enabling efficient execution on Sparse Tensor Cores (SpTCs). By relaxing discrete support selection through continuous reparameterization, the method jointly optimizes quantized weights and supports a router‑weighted reconstruction objective, achieving up to 4.35 percentage‑point gains in joint sparse‑quantization accuracy while retaining 96.09% of the original model’s performance. A custom grouped sparse GEMM kernel further boosts inference speed, outperforming NVIDIA’s baseline by up to 1.65× and reducing latency by up to 4.03× on B200 GPUs.
arXiv:2608. 01536v1 Announce Type: cross Abstract: Large Language Models (LLMs) increasingly rely on sparsity to reduce inference cost, but most prior work targets a single sparsity source-either weight or activation-and optimizes for batched multi-user inference.
Celty introduces a co-designed sparse format, GPU kernel, and SIMT microarchitecture to efficiently handle Sparse Matrix‑Sparse Vector (SpMSpV) workloads in large language model inference. Its Run‑Length Compressed CSC (RLC‑CSC) format allows vectorized loading of compressed weight columns and skips memory accesses by exploiting both weight pruning and activation sparsity. The Celty Sparse SIMT Core adds a pipelined RLC decoder that eliminates software index reconstruction and uses local registers for conflict‑free accumulation, achieving up to 5.3× speedup over cuBLAS at 70% dual‑sparsity.
arXiv:2608. 12629v1 Announce Type: new Abstract: GPU kernel agents and GPU programming languages have advanced separately, leaving expert kernels difficult to reproduce.
KernelOPT is a multi‑agent system that optimizes GPU kernels generated by compilers like PyTorch Inductor by treating compiled models as structured artifacts. It preserves vendor library calls and focuses on Triton sub‑kernels, using five profiling‑guided LLM agents and a four‑gate verification cascade to ensure correctness and performance before re‑stitching the model. On 250 KernelBench problems, KernelOPT achieves geometric mean speedups of 1.40×, 1.15×, and 1.07× over torch.compile at three optimization levels.
TrainBridge is an agentic approach that transfers optimization guidance from GPU training to Ascend NPU training. It extracts optimization intents from highly optimized GPU execution and uses them to rewrite NPU programs, then refines these rewrites through iterative profiling. Experiments show that TrainBridge yields an average 1.7× speedup across four workloads and nearly 2.0× speedup on 27 Modded‑NanoGPT speedrun records.
arXiv:2607. 08786v1 Announce Type: cross Abstract: With the growing deployment of large language models (LLMs), LLM inference cost has become a key challenge.
KernelOPT is a multi-agent system that optimizes GPU kernels generated by compilers like PyTorch Inductor by treating compiled models as structured artifacts. It preserves vendor library calls and focuses on Triton sub-kernels, using five profiling-guided LLM agents and a four-gate verification cascade to filter and validate candidates. On 250 KernelBench problems, KernelOPT achieves geometric mean speedups of 1.40×, 1.15×, and 1.07× over torch.compile at different optimization levels.
The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.