The paper investigates Graph Transformers (GTs) that use Graph Neural Network (GNN)-based positional encodings. By analyzing GTs through manifold limit models, the authors establish a theoretical link between GTs and Manifold Neural Networks (MNNs), showing that GTs inherit transferability guarantees from their positional encodings. Experiments on standard graph benchmarks confirm that GTs scale comparably to GNNs, and a real‑world shortest‑path estimation task demonstrates their practical efficiency on large terrains.
arXiv:2602.15239v3 Announce Type: replace Abstract: Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for...
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging.
arXiv:2602. 01553v3 Announce Type: replace-cross Abstract: Link prediction is a core challenge in graph machine learning, demanding models that capture rich and complex topological dependencies.
arXiv:2606. 25293v1 Announce Type: new Abstract: Positional encodings (PEs) are essential for Transformers.
HyPE-GT introduces a framework that generates learnable hyperbolic positional encodings for Graph Transformers, enabling the capture of complex hierarchical relationships in graph-structured data. Unlike traditional Euclidean encodings, HyPE’s hyperbolic encodings can be selected to suit specific downstream tasks and help mitigate oversmoothing in deep Graph Neural Networks. Experiments on molecular benchmarks and large-scale Open Graph Benchmark datasets demonstrate improved performance, while additional tests on Coauthor and Copurchase networks confirm HyPE’s effectiveness in controlling oversmoothing.
The paper shows that a plain Transformer can serve as an effective graph learner by adding three lightweight modifications: simplified L₂ attention, adaptive RMS normalization, and an MLP-based positional encoding stem. These changes preserve token magnitude and enable the model to achieve high expressivity on graph benchmarks, outperforming more complex graph transformer variants. The results suggest that plain Transformers can act as a unified backbone for multimodal learning across language, vision, and graph domains.
arXiv:2502. 16533v3 Announce Type: replace-cross Abstract: Graph Transformers (GTs) have demonstrated a strong capability in modeling graph structures by addressing the intrinsic limitations of graph neural networks (GNNs), such as over-smoothing and over-squashing.
arXiv:2509. 22259v4 Announce Type: replace-cross Abstract: We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data.
arXiv:2607. 27767v1 Announce Type: new Abstract: Graph neural networks (GNNs) can operate on large graphs but become infrastructure-sensitive at the scale of millions of nodes and typically require scalable training techniques for even larger graphs.
The paper introduces a Position Encoding-Based Deformable Spatial Aggregation Module (PEBDSAM) that uses a deformable mechanism in position space to identify relevant nodes beyond first‑order neighbors, thereby addressing over‑smoothing, over‑compression, limited receptive fields, and noise from heterophilous graphs. Diagnostic experiments revealed that current offsets are ineffective, yet performance still improves, leading to a streamlined version called PEBSAM and a faster variant, PEBSAM‑Speed. The module is plug‑and‑play and can be integrated into GCN, GAT, GIN, and GraphSAGE, achieving strong results on both homophilous and heterophilous datasets.
arXiv:2505. 13087v2 Announce Type: replace-cross Abstract: We propose a novel benchmarking methodology for graph neural networks (GNNs) based on the graph alignment problem, a combinatorial optimization task that generalizes graph isomorphism by aligning two unlabeled graphs to maximize overlapping edges.
arXiv:2503. 01805v3 Announce Type: replace-cross Abstract: Transformers have revolutionized the field of machine learning.