Hugging Face Trending Papers
Jun 24
Communicability-Inspired Positional Encoding (CIPE)
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging.
Enhancing Transformer-based Routing by Encoding Distance via Relative Positional Encoding
arXiv:2607. 18909v1 Announce Type: new Abstract: This paper explores Relative Positional Encoding (RPE) as an additive bias in Transformer architectures to solve the Team Orienteering Problem.
Related stories
arXiv Machine Learning
Jun 25
Communicability-Inspired Positional Encoding (CIPE)
arXiv:2606. 25293v1 Announce Type: new Abstract: Positional encodings (PEs) are essential for Transformers.
arXiv Machine Learning
Oct 16
Size-Transferability of Graph Transformers with Convolutional Positional Encodings
The paper investigates Graph Transformers (GTs) that use Graph Neural Network (GNN)-based positional encodings. By analyzing GTs through manifold limit models, the authors establish a theoretical link between GTs and Manifold Neural Networks (MNNs), showing that GTs inherit transferability guarantees from their positional encodings. Experiments on standard graph benchmarks confirm that GTs scale comparably to GNNs, and a real‑world shortest‑path estimation task demonstrates their practical efficiency on large terrains.
arXiv Machine Learning
Sep 15
Size Transferability of Graph Transformers with Convolutional Positional Encodings
arXiv:2602.15239v3 Announce Type: replace Abstract: Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for...
arXiv AI
Jun 26
Rotary Position Encodings for Graphs
arXiv:2509. 22259v4 Announce Type: replace-cross Abstract: We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data.
arXiv AI
Jul 15
Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task
arXiv:2510. 18315v2 Announce Type: replace-cross Abstract: We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps.
arXiv Machine Learning
Aug 10
Graph Machine: Exploring Edge Mechanisms as an Inductive Bias
arXiv:2608. 06834v1 Announce Type: new Abstract: Transformers provide a powerful architecture for global content-based matching, but reasoning problems may benefit from a stronger inductive bias toward iterative traversal of latent relations.
arXiv AI
Oct 21
Scaling Graph Reasoning in Language Models with Learnable Graph Positional Encodings
arXiv:2610.27439v1 Announce Type: cross Abstract: Solving reasoning tasks supported on graphs typically requires providing a Language Reasoning Model (LRM) with a serialized description of the graph....
arXiv AI
Jun 4
Do Transformers Need Three Projections? Systematic Study of QKV Variants
arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.
arXiv Machine Learning
Sep 30
Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
The paper investigates how the choice of graph tokenization affects transformer expressivity. It analyzes three tokenization families—spectral, random‑walk, and adjacency—showing that each induces different depth requirements and that some tokenizations are inherently lossy or ill‑conditioned for certain tasks. The authors prove lower bounds and impossibility results for converting between tokenizations and validate these findings with experiments on synthetic and real‑world data.
arXiv Machine Learning
Aug 19
HyPE-GT: where Graph Transformers meet Hyperbolic Positional Encodings
HyPE-GT introduces a framework that generates learnable hyperbolic positional encodings for Graph Transformers, enabling the capture of complex hierarchical relationships in graph-structured data. Unlike traditional Euclidean encodings, HyPE’s hyperbolic encodings can be selected to suit specific downstream tasks and help mitigate oversmoothing in deep Graph Neural Networks. Experiments on molecular benchmarks and large-scale Open Graph Benchmark datasets demonstrate improved performance, while additional tests on Coauthor and Copurchase networks confirm HyPE’s effectiveness in controlling oversmoothing.
arXiv AI
Jun 29
Symmetry-Aware Transformer Training for Automated Planning
arXiv:2508. 07743v2 Announce Type: replace Abstract: While transformers excel in many settings, their application in the field of automated planning is limited.