Conic Certification of Transformer Attention

The paper introduces a conic programming framework for certifying transformer attention, providing guarantees on how far an attention head’s output can shift under input perturbations. It shows that the main challenge lies in the bilinear interaction between varying queries and keys, making certification NP-hard over boxes and inapproximable when logits can shift by a constant. The authors present an exponential cone moment relaxation that yields a quadratic error bound in the perturbation radius, and demonstrate that this approach outperforms linear bound propagation on GPT‑2, achieving tighter bounds and matching predicted second‑order accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms