The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Attention is central to modern AI models for language, vision, and scientific modeling. However, direct evaluation compares every query with every key, requiring quadratic time in the context length,...
arXiv:2605.23751v2 Announce Type: replace Abstract: We revisit the I/O complexity of attention in large language models. Given query-key-value matrices $Q,K,V\in\mathbb{R}^{n\times d}$, and a machine...
arXiv:2609.06327v2 Announce Type: replace-cross Abstract: A query-oblivious coreset for a softmax-attention head is a subset of the key-value pairs whose attention output is within $\varepsilon$ of t...
The paper introduces a conic programming framework for certifying transformer attention, providing guarantees on how far an attention head’s output can shift under input perturbations. It shows that the main challenge lies in the bilinear interaction between varying queries and keys, making certification NP-hard over boxes and inapproximable when logits can shift by a constant. The authors present an exponential cone moment relaxation that yields a quadratic error bound in the perturbation radius, and demonstrate that this approach outperforms linear bound propagation on GPT‑2, achieving tighter bounds and matching predicted second‑order accuracy.
arXiv:2608. 28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention?
arXiv:2610. 20637v1 Announce Type: new Abstract: We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one outer product per token and independent of context length $N$.