arXiv Machine Learning

Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers

The paper investigates how the choice of graph tokenization affects transformer expressivity. It analyzes three tokenization families—spectral, random‑walk, and adjacency—showing that each induces different depth requirements and that some tokenizations are inherently lossy or ill‑conditioned for certain tasks. The authors prove lower bounds and impossibility results for converting between tokenizations and validate these findings with experiments on synthetic and real‑world data.

arXiv Machine Learning

Aug 28

Plain Transformers Can be Powerful Graph Learners

The paper shows that a plain Transformer can serve as an effective graph learner by adding three lightweight modifications: simplified L₂ attention, adaptive RMS normalization, and an MLP-based positional encoding stem. These changes preserve token magnitude and enable the model to achieve high expressivity on graph benchmarks, outperforming more complex graph transformer variants. The results suggest that plain Transformers can act as a unified backbone for multimodal learning across language, vision, and graph domains.

By Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip H. S. Torr, Mark Coates

llms multimodal benchmarks

arXiv AI

Jul 28

Plain Transformers are Surprisingly Powerful Link Predictors

arXiv:2602. 01553v3 Announce Type: replace-cross Abstract: Link prediction is a core challenge in graph machine learning, demanding models that capture rich and complex topological dependencies.

By Quang Truong, Yu Song, Donald Loveland, Mingxuan Ju, Tong Zhao, Neil Shah, Jiliang Tang

llms rag benchmarks

arXiv AI

Jul 28

A Survey of Graph Transformers: Architectures, Theories and Applications

arXiv:2502. 16533v3 Announce Type: replace-cross Abstract: Graph Transformers (GTs) have demonstrated a strong capability in modeling graph structures by addressing the intrinsic limitations of graph neural networks (GNNs), such as over-smoothing and over-squashing.

By Chaohao Yuan, Kangfei Zhao, Ercan Engin Kuruoglu, Liang Wang, Tingyang Xu, Wenbing Huang, Deli Zhao, Hong Cheng, Yu Rong

llms nlp

arXiv Machine Learning

Aug 19

HyPE-GT: where Graph Transformers meet Hyperbolic Positional Encodings

HyPE-GT introduces a framework that generates learnable hyperbolic positional encodings for Graph Transformers, enabling the capture of complex hierarchical relationships in graph-structured data. Unlike traditional Euclidean encodings, HyPE’s hyperbolic encodings can be selected to suit specific downstream tasks and help mitigate oversmoothing in deep Graph Neural Networks. Experiments on molecular benchmarks and large-scale Open Graph Benchmark datasets demonstrate improved performance, while additional tests on Coauthor and Copurchase networks confirm HyPE’s effectiveness in controlling oversmoothing.

By Kushal Bose, Swagatam Das

llms benchmarks safety

arXiv Machine Learning

Jun 3

When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

arXiv:2606. 03712v1 Announce Type: new Abstract: Graph Language Models (GLMs) have become a promising direction for adapting Large Language Models (LLMs) to graph learning tasks.

By Ding Zhang, Runtao Zhou, Wenqing Zheng, Rizal Fathony, Bayan Bruss, Chirag Agarwal

llms efficiency multimodal safety

arXiv AI

Jun 18

Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers

arXiv:2503. 01805v3 Announce Type: replace-cross Abstract: Transformers have revolutionized the field of machine learning.

By Gilad Yehudai, Clayton Sanford, Maya Bechler-Speicher, Orr Fischer, Ran Gilad-Bachrach, Amir Globerson

llms rag

arXiv AI

Jun 2

Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learning

arXiv:2508. 06588v3 Announce Type: replace-cross Abstract: Vector Quantization (VQ) has recently emerged as a promising approach for learning compressed and discrete representations for graph-structured data.

By Zian Zhai, Fan Li, Xingyu Tan, Xiaoyang Wang, Wenjie Zhang

efficiency benchmarks

arXiv Machine Learning

Sep 15

Size Transferability of Graph Transformers with Convolutional Positional Encodings

arXiv:2602.15239v3 Announce Type: replace Abstract: Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for...

By Javier Porras-Valenzuela, Zhiyang Wang, Teresa Shang, Yusu Wang, Alejandro Ribeiro

llms benchmarks

arXiv AI

Jun 16

Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

arXiv:2505. 18227v4 Announce Type: replace-cross Abstract: In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks.

By Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, Marinka Zitnik

llms rag agents reinforcement-learning efficiency multimodal safety

arXiv Machine Learning

Jul 21

A Weisfeiler-Leman Characterization of Global-Attention Graph Transformers for Mixed-Integer Linear Programs

arXiv:2607. 17570v1 Announce Type: new Abstract: Graph foundation models (GFMs) with global attention are increasingly used to represent mixed-integer linear programs (MILPs), aiming to capture structure beyond the locality of standard graph neural networks.

By Md Abrar Jahin, Craig A. Knoblock, Jay Pujara

llms rag

arXiv AI

Jul 14

Graph Optimization Foundation Model: Tokenizing Graph via A Language-Model Paradigm

arXiv:2509. 24256v2 Announce Type: replace-cross Abstract: The pretrain-transfer paradigm, which underpins the success of large language models (LLMs), has demonstrated the immense power of creating foundation models that learn generalizable representations from vast datasets.

By Yunhao Liang, Pujun Zhang, Yuan Qu, Jingyuan Yang, Shaochong Lin, Zuo-jun Max Shen

llms

arXiv Machine Learning

Aug 4

Agentic Graph Token Reasoning

arXiv:2608. 00542v1 Announce Type: new Abstract: Graphs model relational data throughout science and industry, from citation networks to product co-purchase graphs.

By Zhuoyi Peng, Yi Yang

llms agents nlp fine-tuning

Related stories