The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2611.06055v1 Announce Type: cross Abstract: Attention is central to modern AI models for language, vision, and scientific modeling. However, direct evaluation compares every query with every ke...
arXiv:2605.23751v2 Announce Type: replace Abstract: We revisit the I/O complexity of attention in large language models. Given query-key-value matrices $Q,K,V\in\mathbb{R}^{n\times d}$, and a machine...
arXiv:2608. 28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention?
arXiv:2609.06327v2 Announce Type: replace-cross Abstract: A query-oblivious coreset for a softmax-attention head is a subset of the key-value pairs whose attention output is within $\varepsilon$ of t...
The paper introduces a conic programming framework for certifying transformer attention, providing guarantees on how far an attention head’s output can shift under input perturbations. It shows that the main challenge lies in the bilinear interaction between varying queries and keys, making certification NP-hard over boxes and inapproximable when logits can shift by a constant. The authors present an exponential cone moment relaxation that yields a quadratic error bound in the perturbation radius, and demonstrate that this approach outperforms linear bound propagation on GPT‑2, achieving tighter bounds and matching predicted second‑order accuracy.
arXiv:2608.28150v2 Announce Type: replace Abstract: How much matrix rank is required to preserve every bounded value output of normalized softmax attention? We study the unrestricted maximum-row-\(\e...