HyPE-GT introduces a framework that generates learnable hyperbolic positional encodings for Graph Transformers, enabling the capture of complex hierarchical relationships in graph-structured data. Unlike traditional Euclidean encodings, HyPE’s hyperbolic encodings can be selected to suit specific downstream tasks and help mitigate oversmoothing in deep Graph Neural Networks. Experiments on molecular benchmarks and large-scale Open Graph Benchmark datasets demonstrate improved performance, while additional tests on Coauthor and Copurchase networks confirm HyPE’s effectiveness in controlling oversmoothing.
By Kushal Bose, Swagatam Das
arXiv:2607. 01553v1 Announce Type: cross Abstract: Transformers have become general-purpose architectures, but their all-to-all self-attention is poorly matched to graph data, whose interactions are sparse, structured and multi-scale.
By Leyan Li, Rennong Yang, Zhenxing Zhang, Liping Hu
arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.
By Yaniv Livertovsky, Shahar Somin, Gonen Singer
arXiv:2607. 27303v1 Announce Type: new Abstract: Temporal heterogeneous graphs offer a natural abstraction for dynamic relational systems in which diverse node and relation types co-exist and evolve over time.
By Yixin Peng, Diego Collarana, Er Jin, Stefan Decker
The paper introduces OASIS, a method designed to stabilize dual‑normalized attention‑residual architectures by employing explicit null routing and token‑to‑depth null coupling. OASIS mitigates attention sinks and activation outliers, improving low‑bit quantization performance across several language‑model backbones. Empirical results show significant reductions in attention norms and perplexity, with notable gains on long‑context benchmarks.
By Haozheng Luo, Haoran Dai, Jingyuan Huang, Shaoyang Zhang, Xi Chen, Eric Hanchen Jiang, Yijiang Li, Chenghao Qiu, Chenwei Xu, Zhenyu Pan, Haotian Zhang, Binghui Wang, Yan Chen
arXiv:2607. 17570v1 Announce Type: new Abstract: Graph foundation models (GFMs) with global attention are increasingly used to represent mixed-integer linear programs (MILPs), aiming to capture structure beyond the locality of standard graph neural networks.
By Md Abrar Jahin, Craig A. Knoblock, Jay Pujara
arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
By Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu
arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.
By Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan
arXiv:2601.06787v2 Announce Type: replace
Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...
By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Taesup Kim
arXiv:2607. 22781v1 Announce Type: cross Abstract: High task performance does not show whether a model retains prediction-relevant structural information in its internal representation.
By Minwoo Yu, Young-guk Ha
The paper shows that a plain Transformer can serve as an effective graph learner by adding three lightweight modifications: simplified L₂ attention, adaptive RMS normalization, and an MLP-based positional encoding stem. These changes preserve token magnitude and enable the model to achieve high expressivity on graph benchmarks, outperforming more complex graph transformer variants. The results suggest that plain Transformers can act as a unified backbone for multimodal learning across language, vision, and graph domains.
By Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip H. S. Torr, Mark Coates
arXiv:2607. 20484v1 Announce Type: new Abstract: Large Language Models (LLMs) are fundamentally limited by representation collapse, a bottleneck that severely degrades long-context performance.
By Yiheng Tao, Kaiwen Cheng, Yao Lu, Chang Liu, Jie Chen