The paper introduces a new graphical notation, adapted from Penrose tensor notation, to design and represent interpretable AI architectures. Unlike symbolic equations or probabilistic models, this notation provides a global view of an architecture while directly mapping to PyTorch einsum code. The authors demonstrate its use on several interpretable models and on the Steerling-8B language model, revealing structural insights and enabling concise code generation.
arXiv:2604. 07242v3 Announce Type: replace Abstract: Despite deep learning models running well-defined mathematical functions, we lack a formal mathematical framework for describing model architectures.
By Vincent Abbott, Gioele Zardini
arXiv:2608.29034v1 Announce Type: cross
Abstract: A wide range of methods have been proposed for interpreting language models, delivering important insights into their inner workings. However, differ...
By Zhang Enyan, R. Thomas McCoy
The paper discusses tensorizing neural networks by reshaping dense weight matrices into higher-order tensors and approximating them with low-rank tensor network decompositions. This approach offers promising model compression and introduces bond indices that create new latent spaces, potentially enhancing interpretability. Despite encouraging empirical results, tensorized neural networks remain underused, and the authors call for more research to address practical scaling and adoption challenges.
By Safa Hamreras, Sukhbinder Singh, Rom\'an Or\'us
arXiv:2608. 01633v1 Announce Type: new Abstract: Large language models (LLMs) enable neural architecture search (NAS) directly over executable neural network programs.
By Zhen Liu, Wanqi Zhou, Shuanghao Bai, Yuhan Liu, Jinjun Wang, Jingwen Fu
Neuton is a framework that automatically converts neuron‑level abstract interpreter specifications into tensor‑level executables, enabling the creation of new DNN certifiers without manual tensor programming. It introduces a stack‑based intermediate representation with shape analysis, domain‑specific rewrite rules for efficient tensor operations, and a novel sparse tensor format (gBCSR) tailored to certifier sparsity. The authors demonstrate that Neuton preserves neuron‑level semantics and evaluate it on eight certifiers across 32 DNN benchmarks, compiling four existing certifiers and producing executable implementations for four others, revealing new precision‑runtime trade‑offs such as SkipPoly matching DeepPoly’s precision while often running faster.
By Avaljot Singh, Yasmin Chandini Sarita, Aditya Mishra, Gagandeep Singh, Charith Mendis
This survey reviews tensor methods applied to large language models, framing them through a seven‑stage lifecycle (tokenization, embeddings, pre‑training, adaptation, compression, inference, interpretability) and a component view (embeddings, attention, feed‑forward networks). It offers unified notation, theoretical foundations, and comparative analyses of tensorization strategies for Transformer components, while highlighting evaluation protocol differences and model scale effects. The paper also introduces a new metric, ρ_gap, to quantify the gap between theoretical memory savings and actual system‑level speedup, and connects tensor techniques to related efficiency and probabilistic methods.
By Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki
The paper presents a unified theoretical framework for analyzing and constructing deep neural networks by explicitly modeling tensor operations, which are often abstracted in existing theories. This framework allows researchers to study how architectural complexity has evolved over the past 40 years and to automatically generate new architectures using novel tensor operations. The authors also release a dataset of over 3,000 higher‑complexity architectures that have not yet been explored, providing a resource for future research.
By Nicholas J. Cooper, Fran\c{c}ois G. Meyer, Michael L. Roberts, Carlos Zapata-Carratal\'a, Lijun Chen, Danna Gurari
Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is unde...
arXiv:2606. 19279v1 Announce Type: new Abstract: Neurosymbolic semantics is fragmented: classical, fuzzy, probabilistic and neural systems each define truth by their own inductive rules.
By Daniel Romero Schellhorn, Till Mossakowski, Bj\"orn Gehrke
arXiv:2608. 11136v1 Announce Type: new Abstract: Logic Tensor Networks (LTN) provide a neurosymbolic framework in which first-order logic is interpreted through tensor operations, enabling logical constraints to be integrated with differentiable learning.
By Davide Rinaldi, Luciano Serafini
arXiv:2609.13198v1 Announce Type: new
Abstract: One of the pivotal recent challenges in neural network interpretability is polysemanticity, where a single neuron is activated by multiple, often unrel...
By Sehyun Lee, Dahee Kwon, Damin Lee, Jaesik Choi