← Back to all news
arXiv AI By Aakash Sarkar, Ananya Mukherjee, Motoshige Sato

Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention

Read the original on arXiv AI →

The paper investigates how treating each entity in a graph or language model as having its own coordinate frame—viewed as a symmetry—constrains the form of attention mechanisms. It shows that with orthogonal transports, invariant attention scores depend only on norms and transported inner products, while equivariant messages lie in the span of the entity vectors and their transported counterparts. The authors demonstrate that standard attention corresponds to a flat shared frame, and that incorporating per‑entity frames allows relations to be represented as operators that compose via parallel transport, yielding empirical gains on synthetic relational tasks and natural language benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 24

Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings

The paper argues that meaning identity—whether two sentences convey the same idea after wording changes—is not encoded in the geometry of independently produced sentence embeddings. Experiments on frozen off‑the‑shelf encoders and language models show that identity can only be reliably computed when both sentences are processed together in a single forward pass, yielding high accuracy (0.90–0.96) on PAWS‑X, whereas independent embeddings or simple fusion methods perform near chance. Even advanced bi‑encoder fine‑tuning improves performance on PAWS but fails to generalize to other similarity tasks, underscoring that identity is a cheap computed operator rather than a property of individual sentence vectors.

By Jiaqi Deng
llmsragfine-tuning
More like this →
arXiv Computation and Language
Sep 10

Through the Looking Glass: Directly Reading and Writing Transformers

The paper investigates how many transformer components influence a token prediction by measuring the absolute contribution of each unit and channel to the logit. It finds that thousands of components contribute to a single prediction, yet a small subset—often just dozens—carries the majority of the predictive mass. Across models ranging from 124 M to 7 B parameters, the proportion of the model involved in a prediction remains around one to three percent, independent of size, and the study demonstrates that specific components can be directly read and written to modify model behavior without additional training.

By Mark Oskin
llmsrag
More like this →