The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
This survey reviews tensor methods applied to large language models, framing them through a seven‑stage lifecycle (tokenization, embeddings, pre‑training, adaptation, compression, inference, interpretability) and a component view (embeddings, attention, feed‑forward networks). It offers unified notation, theoretical foundations, and comparative analyses of tensorization strategies for Transformer components, while highlighting evaluation protocol differences and model scale effects. The paper also introduces a new metric, ρ_gap, to quantify the gap between theoretical memory savings and actual system‑level speedup, and connects tensor techniques to related efficiency and probabilistic methods.
arXiv:2606. 03465v1 Announce Type: cross Abstract: Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints.
Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures.
arXiv:2607.12550v3 Announce Type: replace-cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, a...
arXiv:2608. 13426v1 Announce Type: cross Abstract: Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications.
TensorDex is a lossless, tensor‑centric storage system designed to reduce the storage footprint of large language models by performing delta compression at the tensor level. It decomposes models into individual tensors, uses compact fingerprints to predict which tensors can be compressed together, and incrementally clusters tensors to select effective reference bases as new models arrive. In experiments on 2,890 Hugging Face models, TensorDex cuts storage usage by 70.5%, outperforms the previous best system by 37%, and achieves compression and decompression speeds of 22.9 GB/s and 28.4 GB/s, respectively.