arXiv AI September 1, 2026 By Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

This survey reviews tensor methods applied to large language models, framing them through a seven‑stage lifecycle (tokenization, embeddings, pre‑training, adaptation, compression, inference, interpretability) and a component view (embeddings, attention, feed‑forward networks). It offers unified notation, theoretical foundations, and comparative analyses of tensorization strategies for Transformer components, while highlighting evaluation protocol differences and model scale effects. The paper also introduces a new metric, ρ_gap, to quantify the gap between theoretical memory savings and actual system‑level speedup, and connects tensor techniques to related efficiency and probabilistic methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

4d ago

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is unde...

llms rag nlp safety

arXiv AI

Jun 3

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

arXiv:2606. 03465v1 Announce Type: cross Abstract: Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints.

By Artur Zagitov, Alexander Miasnikov, Maxim Krutikov, Vladimir Aletov, Gleb Molodtsov, Nail Bashirov, Artem Tsedenov, Aleksandr Beznosikov

llms

Hugging Face Trending Papers

Jun 2

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures.

llms

arXiv Computation and Language

Aug 25

A JoLT for the KV cache: Near-lossless KV cache compression via joint Lagrangian allocation of Tucker ranks and a rotated residual for llms

arXiv:2607.12550v3 Announce Type: replace-cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, a...

By Rahul Krishnan, Volker Schulz

llms efficiency

arXiv AI

Aug 14

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

arXiv:2608. 13426v1 Announce Type: cross Abstract: Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications.

By Zixuan Lan, Yanhong Li, Jiawei Zhou

llms multimodal benchmarks

arXiv AI

Jun 16

Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

arXiv:2505. 18227v4 Announce Type: replace-cross Abstract: In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks.

By Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, Marinka Zitnik

llms rag agents reinforcement-learning efficiency multimodal safety

One email a morning, machine-written

Related stories

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

A JoLT for the KV cache: Near-lossless KV cache compression via joint Lagrangian allocation of Tucker ranks and a rotated residual for llms

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality