← Back to all news
arXiv AI

Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention

The paper investigates how treating each entity in a graph or language model as having its own coordinate frame—viewed as a symmetry—constrains the form of attention mechanisms. It shows that with orthogonal transports, invariant attention scores depend only on norms and transported inner products, while equivariant messages lie in the span of the entity vectors and their transported counterparts. The authors demonstrate that standard attention corresponds to a flat shared frame, and that incorporating per‑entity frames allows relations to be represented as operators that compose via parallel transport, yielding empirical gains on synthetic relational tasks and natural language benchmarks.

Related stories

arXiv Computation and Language
Sep 24

Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings

The paper argues that meaning identity—whether two sentences convey the same idea after wording changes—is not encoded in the geometry of independently produced sentence embeddings. Experiments on frozen off‑the‑shelf encoders and language models show that identity can only be reliably computed when both sentences are processed together in a single forward pass, yielding high accuracy (0.90–0.96) on PAWS‑X, whereas independent embeddings or simple fusion methods perform near chance. Even advanced bi‑encoder fine‑tuning improves performance on PAWS but fails to generalize to other similarity tasks, underscoring that identity is a cheap computed operator rather than a property of individual sentence vectors.

By Jiaqi Deng
llmsragfine-tuning
More like this →
arXiv Computation and Language
Sep 10

Through the Looking Glass: Directly Reading and Writing Transformers

The paper investigates how many transformer components influence a token prediction by measuring the absolute contribution of each unit and channel to the logit. It finds that thousands of components contribute to a single prediction, yet a small subset—often just dozens—carries the majority of the predictive mass. Across models ranging from 124 M to 7 B parameters, the proportion of the model involved in a prediction remains around one to three percent, independent of size, and the study demonstrates that specific components can be directly read and written to modify model behavior without additional training.

By Mark Oskin
llmsrag
More like this →
arXiv AI
Jul 23

Geometric Attention: A Regime-Explicit Operator Semantics for Transformer Attention

arXiv:2601. 11618v2 Announce Type: replace-cross Abstract: Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied).

By Luis Rosario Freytes
llms
More like this →
arXiv Machine Learning
Jun 11

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

arXiv:2605. 04893v2 Announce Type: replace Abstract: When a language model processes a hallucinated response, its attention routing tends to fail in one of two shapes: over-concentrating on a narrow set of positions, or spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal.

By Dominik Dahlem, Diego Maniloff, Mac Misiura
llmsbenchmarks
More like this →