arXiv:2610. 41218v1 Announce Type: new Abstract: Why does length generalization (LG) depend so sharply on the positional encoding (PE)?
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
The paper investigates how treating each entity in a graph or language model as having its own coordinate frame—viewed as a symmetry—constrains the form of attention mechanisms. It shows that with orthogonal transports, invariant attention scores depend only on norms and transported inner products, while equivariant messages lie in the span of the entity vectors and their transported counterparts. The authors demonstrate that standard attention corresponds to a flat shared frame, and that incorporating per‑entity frames allows relations to be represented as operators that compose via parallel transport, yielding empirical gains on synthetic relational tasks and natural language benchmarks.
arXiv:2608.20647v1 Announce Type: new Abstract: Splitting a bidirectional LSTM's contextual representation into a forward-only $F_i$ (strictly a function of tokens $1..i$) and a backward-only $B_i$ (...
arXiv:2609.01129v1 Announce Type: new Abstract: We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under compos...
arXiv:2605. 25889v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models reach high success rates on clean inputs but collapse under small adversarial perturbations: a $16/255$ PGD attack drops OpenVLA-7B's LIBERO success from $95\%$ to under $5\%$.
arXiv:2608. 03620v1 Announce Type: cross Abstract: Activation patching and weight-space ablation both claim a component is causally responsible for a behavior, yet they act on different objects: one forward pass versus the parameters behind every forward pass.