← Back to all news
arXiv Machine Learning By Zicong Huang

Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States

Read the original on arXiv Machine Learning →

arXiv:2610. 20637v1 Announce Type: new Abstract: We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly $n$ recurrent states, each updated by one rank-one outer product per token and independent of context length $N$.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
1d ago

Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds

arXiv:2611.06055v1 Announce Type: cross Abstract: Attention is central to modern AI models for language, vision, and scientific modeling. However, direct evaluation compares every query with every ke...

By Zixuan Huang (State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China, School of Physics, Peking University, Beijing, China), Guozhang Chen (State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China, National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China)
More like this →
arXiv Computation and Language
2d ago

Conic Certification of Transformer Attention

The paper introduces a conic programming framework for certifying transformer attention, providing guarantees on how far an attention head’s output can shift under input perturbations. It shows that the main challenge lies in the bilinear interaction between varying queries and keys, making certification NP-hard over boxes and inapproximable when logits can shift by a constant. The authors present an exponential cone moment relaxation that yields a quadratic error bound in the perturbation radius, and demonstrate that this approach outperforms linear bound propagation on GPT‑2, achieving tighter bounds and matching predicted second‑order accuracy.

By Masoud Ataei
llms
More like this →