TensorDex is a lossless, tensor‑centric storage system designed to reduce the storage footprint of large language models by performing delta compression at the tensor level. It decomposes models into individual tensors, uses compact fingerprints to predict which tensors can be compressed together, and incrementally clusters tensors to select effective reference bases as new models arrive. In experiments on 2,890 Hugging Face models, TensorDex cuts storage usage by 70.5%, outperforms the previous best system by 37%, and achieves compression and decompression speeds of 22.9 GB/s and 28.4 GB/s, respectively.
Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is unde...
The paper introduces JoLT, a training‑free compressor that jointly allocates rank and precision for key‑value (KV) cache compression in long‑context language models. JoLT treats grouped prefill caches as fourth‑order tensors, applies partial Tucker decomposition along token and feature modes, and uses a rotated low‑bit quantizer for residuals, all governed by a single Lagrangian dual under a global byte constraint. Across five models from four architecture families, JoLT achieves 2–3× compression with less than 0.2% perplexity loss, and near‑lossless retrieval accuracy on LLaMA‑3.1‑8B at 64K context up to 3× compression.
This survey reviews tensor methods applied to large language models, framing them through a seven‑stage lifecycle (tokenization, embeddings, pre‑training, adaptation, compression, inference, interpretability) and a component view (embeddings, attention, feed‑forward networks). It offers unified notation, theoretical foundations, and comparative analyses of tensorization strategies for Transformer components, while highlighting evaluation protocol differences and model scale effects. The paper also introduces a new metric, ρ_gap, to quantify the gap between theoretical memory savings and actual system‑level speedup, and connects tensor techniques to related efficiency and probabilistic methods.
NeuralZip is a method for lossless compression of floating‑point model checkpoints that pre‑computes a reusable setup based on exponent statistics. By grouping tensor chunks with similar exponent distributions, sharing Huffman codes, and packing recurring exponent tuples, it achieves 1.81–21.33× faster compression than baselines while preserving exact bit‑to‑bit reconstruction. The setup can be transferred across compatible architectures and reused during training, reducing GPU memory usage by up to 27.5% without affecting model outputs.
arXiv:2606. 03465v1 Announce Type: cross Abstract: Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints.
arXiv:2605. 01708v3 Announce Type: replace-cross Abstract: Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase.
Lossless compression can reduce the storage and movement of model weights without changing their floating-point values, but repeated statistical analysis and code construction add computational overhe...
arXiv:2607.12550v3 Announce Type: replace-cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, a...
arXiv:2607. 12550v1 Announce Type: new Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference.
Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures.
The paper introduces Losslessly Compressed Tensors (LCT), a GPU‑friendly framework that repeatedly encodes and decodes tensors during training. LCT uses Huffman coding with precomputed codebooks to exploit redundancy in floating‑point data, achieving up to a 1.455× compression ratio. Experiments on models such as NanoGPT and Qwen3‑4B show memory savings of 24.6% and 23.8% respectively, enabling larger context lengths with only modest increases in step time.
arXiv:2604.09107v2 Announce Type: replace-cross Abstract: Modern LLM reinforcement learning (RL) workloads require a high-performance weight transfer system to scale training across heterogeneous com...