Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering
Read the original on arXiv Machine Learning →The paper demonstrates that any Transformer block—comprising attention, LayerNorm, and a GELU MLP—can be exactly compiled into a generic deep neural network using only fixed affine maps, routing, and a small set of nonlinear functions (square, sigmoid, inverse square root, and GELU or its sigmoid approximation). It shows that removing activation‑activation products incurs an $N^2$ cost per head for dense scores, but this can be avoided by using element‑wise operations available on most NPUs. Additionally, the authors prove that exact or conditional‑mean tail‑mass estimation provides an unbiased, low‑error approximation of top‑$k$ attention, and that clustering queries or agents into centroids reduces the number of computed scores from $N^2$ to $Nk$ per head, achieving comparable performance to reference models on DeiT‑Tiny, MambaIRv2‑Light, and Pythia‑70M.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.