NeuralZip is a method for lossless compression of floating‑point model checkpoints that pre‑computes a reusable setup based on exponent statistics. By grouping tensor chunks with similar exponent distributions, sharing Huffman codes, and packing recurring exponent tuples, it achieves 1.81–21.33× faster compression than baselines while preserving exact bit‑to‑bit reconstruction. The setup can be transferred across compatible architectures and reused during training, reducing GPU memory usage by up to 27.5% without affecting model outputs.
By Mart\'in Bravo, Samuel Horv\'ath, Gonzalo Navarro, Andr\'es Abeliuk
Lossless compression can reduce the storage and movement of model weights without changing their floating-point values, but repeated statistical analysis and code construction add computational overhe...
arXiv:2605. 01708v3 Announce Type: replace-cross Abstract: Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase.
By Yipin Guo, Siddharth Joshi
BitSparse introduces a GPU‑resident, lossless compression format for sparse ReLU and ReLU² activations in feed‑forward networks. By integrating custom Triton kernels into an autograd function, it compresses and reconstructs activation caches without loss, supporting BF16 and FP8 formats and offering optional sign‑bit packing to approach theoretical limits. The method reduces per‑token VRAM usage by about 30% for Modded‑NanoGPT and Nemotron‑H, with negligible overhead and a new preallocated buffer system that eliminates host synchronization during training.
By Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu
TensorDex is a lossless, tensor‑centric storage system designed to reduce the storage footprint of large language models by performing delta compression at the tensor level. It decomposes models into individual tensors, uses compact fingerprints to predict which tensors can be compressed together, and incrementally clusters tensors to select effective reference bases as new models arrive. In experiments on 2,890 Hugging Face models, TensorDex cuts storage usage by 70.5%, outperforms the previous best system by 37%, and achieves compression and decompression speeds of 22.9 GB/s and 28.4 GB/s, respectively.
By Tingfeng Lan, Zirui Wang, Yunjia Zheng, Zhaoyuan Su, Juncheng Yang, Yue Cheng
arXiv:2607. 16241v1 Announce Type: cross Abstract: Recent large language models (LLMs) can generate custom CUDA kernels that appear to outperform PyTorch on benchmarks such as KernelBench.
By Yunxiang Zhang (Xiangjun), Ping Yu (Xiangjun), Jianyu Wang (Xiangjun), Max (Xiangjun), Fan, Julian Reed, Azalia Mirhoseini, Will Su
Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning.
Celty introduces a co-designed sparse format, GPU kernel, and SIMT microarchitecture to efficiently handle Sparse Matrix‑Sparse Vector (SpMSpV) workloads in large language model inference. Its Run‑Length Compressed CSC (RLC‑CSC) format allows vectorized loading of compressed weight columns and skips memory accesses by exploiting both weight pruning and activation sparsity. The Celty Sparse SIMT Core adds a pipelined RLC decoder that eliminates software index reconstruction and uses local registers for conflict‑free accumulation, achieving up to 5.3× speedup over cuBLAS at 70% dual‑sparsity.
By Ruokai Yin, Priyadarshini Panda
arXiv:2606. 10445v1 Announce Type: new Abstract: Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup.
By Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari
The paper introduces a hardware-software co‑design framework that compresses Mixture‑of‑Experts (MoE) model weights into low‑precision, hardware‑native sparse representations, enabling efficient execution on Sparse Tensor Cores (SpTCs). By relaxing discrete support selection through continuous reparameterization, the method jointly optimizes quantized weights and supports a router‑weighted reconstruction objective, achieving up to 4.35 percentage‑point gains in joint sparse‑quantization accuracy while retaining 96.09% of the original model’s performance. A custom grouped sparse GEMM kernel further boosts inference speed, outperforming NVIDIA’s baseline by up to 1.65× and reducing latency by up to 4.03× on B200 GPUs.
By Kwanhee Lee, Namhoon Lee, Dan Alistarh
arXiv:2607. 08786v1 Announce Type: cross Abstract: With the growing deployment of large language models (LLMs), LLM inference cost has become a key challenge.
By Tao Lu, Haoyu Wang, Zonghui Wang, Keshen Xiang, Jiaheng Zhang, Wenzhi Chen
arXiv:2610.14277v1 Announce Type: new
Abstract: Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows a...
By Xingyu Xie, Xiaoyong Xie, Zhouchen Lin, Kim-Chuan Toh