Let the Heads Talk: Beyond Diagonal Graph Attention

The paper introduces Topological Attention (Top‑A), a multi‑head attention mechanism that extends standard diagonal edge maps by allowing off‑diagonal, edge‑conditioned communication across attention heads. By isolating the transport primitive through quiver representations, the authors show that standard multi‑head attention only implements diagonal edge maps, whereas Top‑A learns additional cross‑head routes while preserving the original same‑head paths. Experiments on relational reasoning, heterogeneous graph learning, and algorithmic reasoning demonstrate that cross‑head transport is most beneficial when tasks require interaction‑dependent transformations, whereas heterophily alone does not provide a systematic advantage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• diffusion