The paper introduces a graphical notation, adapted from Penrose tensor notation, to design and represent interpretable AI architectures. This notation provides a global view of an architecture and maps directly onto PyTorch einsum code, enabling clear depiction of tensor manipulations. The authors apply the notation to several interpretable models—concept bottlenecks, sparse probes, prototype networks, neural additive models, and mixtures of linear models—and use it to diagram the key components of the Steerling-8B language model, revealing its residual structure and allowing a concise 33‑line PyTorch implementation.
By Pietro Barbiero
arXiv:2604. 07242v3 Announce Type: replace Abstract: Despite deep learning models running well-defined mathematical functions, we lack a formal mathematical framework for describing model architectures.
By Vincent Abbott, Gioele Zardini
arXiv:2608.29034v1 Announce Type: cross
Abstract: A wide range of methods have been proposed for interpreting language models, delivering important insights into their inner workings. However, differ...
By Zhang Enyan, R. Thomas McCoy
arXiv:2608. 01633v1 Announce Type: new Abstract: Large language models (LLMs) enable neural architecture search (NAS) directly over executable neural network programs.
By Zhen Liu, Wanqi Zhou, Shuanghao Bai, Yuhan Liu, Jinjun Wang, Jingwen Fu
Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is unde...
This survey reviews tensor methods applied to large language models, framing them through a seven‑stage lifecycle (tokenization, embeddings, pre‑training, adaptation, compression, inference, interpretability) and a component view (embeddings, attention, feed‑forward networks). It offers unified notation, theoretical foundations, and comparative analyses of tensorization strategies for Transformer components, while highlighting evaluation protocol differences and model scale effects. The paper also introduces a new metric, ρ_gap, to quantify the gap between theoretical memory savings and actual system‑level speedup, and connects tensor techniques to related efficiency and probabilistic methods.
By Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki