← Back to all news
arXiv Machine Learning By Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip H. S. Torr, Mark Coates

Plain Transformers Can be Powerful Graph Learners

Read the original on arXiv Machine Learning →

The paper shows that a plain Transformer can serve as an effective graph learner by adding three lightweight modifications: simplified L₂ attention, adaptive RMS normalization, and an MLP-based positional encoding stem. These changes preserve token magnitude and enable the model to achieve high expressivity on graph benchmarks, outperforming more complex graph transformer variants. The results suggest that plain Transformers can act as a unified backbone for multimodal learning across language, vision, and graph domains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 28

A Survey of Graph Transformers: Architectures, Theories and Applications

arXiv:2502. 16533v3 Announce Type: replace-cross Abstract: Graph Transformers (GTs) have demonstrated a strong capability in modeling graph structures by addressing the intrinsic limitations of graph neural networks (GNNs), such as over-smoothing and over-squashing.

By Chaohao Yuan, Kangfei Zhao, Ercan Engin Kuruoglu, Liang Wang, Tingyang Xu, Wenbing Huang, Deli Zhao, Hong Cheng, Yu Rong
llmsnlp
More like this →
arXiv Machine Learning
3d ago

Size-Transferability of Graph Transformers with Convolutional Positional Encodings

The paper investigates Graph Transformers (GTs) that use Graph Neural Network (GNN)-based positional encodings. By analyzing GTs through manifold limit models, the authors establish a theoretical link between GTs and Manifold Neural Networks (MNNs), showing that GTs inherit transferability guarantees from their positional encodings. Experiments on standard graph benchmarks confirm that GTs scale comparably to GNNs, and a real‑world shortest‑path estimation task demonstrates their practical efficiency on large terrains.

By Javier Porras-Valenzuela, Zhiyang Wang, Teresa Shang, Yusu Wang, Alejandro Ribeiro
llmsbenchmarks
More like this →
arXiv AI
Jun 26

Rotary Position Encodings for Graphs

arXiv:2509. 22259v4 Announce Type: replace-cross Abstract: We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data.

By Isaac Reid, Arijit Sehanobish, Cederik H\"ofs, Bruno Mlodozeniec, Leonhard Vulpius, Federico Barbero, Adrian Weller, Krzysztof Choromanski, Richard E. Turner, Petar Veli\v{c}kovi\'c
llmssafety
More like this →