arXiv:2606. 06160v1 Announce Type: new Abstract: RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product.
Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.
arXiv:2603. 06591v2 Announce Type: replace Abstract: Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks.
arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.
arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
arXiv:2606. 08105v1 Announce Type: new Abstract: When attention concentrates on a single token, a sink, what is the model actually computing?
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.