← Back to all news
arXiv Machine Learning By Andrew O'Desky

On attention heads and bilinear forms

Read the original on arXiv Machine Learning →

The paper investigates the symmetric and antisymmetric components of bilinear forms that arise in the attention heads of large language models. It introduces an orthogonally invariant profile map that projects real bilinear forms onto a three‑dimensional simplex, revealing that the profiles of trained bilinear forms tend to cluster near those of rank‑one bilinear forms. Additionally, the authors prove that for a Zariski‑dense subset of query‑key matrices, the symmetric part of a bilinear form in an attention head can be decomposed into a hyperbolic form plus a zero form.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Oct 5

Emergent Structure in the Marginal Attention Space of Language Models

The paper investigates how post‑softmax attention weights behave across different language models by marginalizing over query positions to create a joint token‑head space. Across more than 60 diverse LLMs, token‑wise marginal attention reveals a text‑intrinsic signal that is conserved across models, while head‑wise marginal attention yields a model‑private signature that is consistent across documents. This insight enables a practical method for estimating per‑head budgets for key‑value cache eviction, achieving competitive performance on standard benchmarks without per‑document recomputation or training.

By Valentino Maiorca, Walter Nelson, Francesco Locatello
llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 16

Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention

arXiv:2608. 30417v2 Announce Type: replace Abstract: We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action.

By T\=ikun \^Ong, Georg B\"okman
More like this →