← Back to all news
arXiv Machine Learning By Maya Bechler-Speicher, Gilad Yehudai, Gil Harari, Clayton Sanford, Amir Globerson, Joan Bruna

Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers

Read the original on arXiv Machine Learning →

The paper investigates how the choice of graph tokenization affects transformer expressivity. It analyzes three tokenization families—spectral, random‑walk, and adjacency—showing that each induces different depth requirements and that some tokenizations are inherently lossy or ill‑conditioned for certain tasks. The authors prove lower bounds and impossibility results for converting between tokenizations and validate these findings with experiments on synthetic and real‑world data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 28

Plain Transformers Can be Powerful Graph Learners

The paper shows that a plain Transformer can serve as an effective graph learner by adding three lightweight modifications: simplified L₂ attention, adaptive RMS normalization, and an MLP-based positional encoding stem. These changes preserve token magnitude and enable the model to achieve high expressivity on graph benchmarks, outperforming more complex graph transformer variants. The results suggest that plain Transformers can act as a unified backbone for multimodal learning across language, vision, and graph domains.

By Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip H. S. Torr, Mark Coates
llmsmultimodalbenchmarks
More like this →
arXiv AI
Jul 28

A Survey of Graph Transformers: Architectures, Theories and Applications

arXiv:2502. 16533v3 Announce Type: replace-cross Abstract: Graph Transformers (GTs) have demonstrated a strong capability in modeling graph structures by addressing the intrinsic limitations of graph neural networks (GNNs), such as over-smoothing and over-squashing.

By Chaohao Yuan, Kangfei Zhao, Ercan Engin Kuruoglu, Liang Wang, Tingyang Xu, Wenbing Huang, Deli Zhao, Hong Cheng, Yu Rong
llmsnlp
More like this →
arXiv Machine Learning
Aug 19

HyPE-GT: where Graph Transformers meet Hyperbolic Positional Encodings

HyPE-GT introduces a framework that generates learnable hyperbolic positional encodings for Graph Transformers, enabling the capture of complex hierarchical relationships in graph-structured data. Unlike traditional Euclidean encodings, HyPE’s hyperbolic encodings can be selected to suit specific downstream tasks and help mitigate oversmoothing in deep Graph Neural Networks. Experiments on molecular benchmarks and large-scale Open Graph Benchmark datasets demonstrate improved performance, while additional tests on Coauthor and Copurchase networks confirm HyPE’s effectiveness in controlling oversmoothing.

By Kushal Bose, Swagatam Das
llmsbenchmarkssafety
More like this →