The paper introduces gated graph attention and a learnable temperature to enhance standard graph attention networks. Gating filters out unreliable feature dimensions, while the temperature parameter adjusts the sharpness of attention coefficient distributions. Experiments on node classification and heterophilic benchmarks show that these modifications improve performance and robustness, with theoretical analysis supporting the observed gains.
By Zhongtian Ma, Hao Wu, Yexin Zhang, Qiaosheng Zhang, Chao Gao
The paper introduces gated graph attention and a learnable temperature to enhance standard graph attention networks. Gated attention filters out unreliable feature dimensions, while the temperature parameter dynamically adjusts the sharpness of attention coefficient distributions. Experiments on both homogeneous and heterophilic heterogeneous benchmarks demonstrate consistent improvements over existing backbones, and controlled noise studies confirm the robustness of these mechanisms.
By Zhongtian Ma, Hao Wu, Yexin Zhang, Qiaosheng Zhang, Zhen Wang
The paper investigates the training dynamics of attention mechanisms in high-dimensional settings, focusing on attention-indexed models that encompass multi-layer and multi-head architectures. It shows that while the loss landscape can be described by a finite set of trace order parameters, the online stochastic gradient descent dynamics involve an infinite hierarchy of matrix moments that can be accurately approximated by a finite truncated system. The study further reveals that the choice of attention parameterization acts as an implicit bias: untied attention can get trapped in uninformative states, whereas tied attention induces symmetry breaking and enables weak recovery with θ(d² log d) samples, and untied attention exhibits a fast-slow dynamic leading to weak recovery when symmetry is broken.
By Yizhou Xu, Margarita Sagitova, Lenka Zdeborov\'a, Florent Krzakala
arXiv:2610.18471v2 Announce Type: replace
Abstract: Weight-space learning treats the weights of neural networks as data, processed by other models called metanetworks. A key challenge is that differe...
By Yonathan Wolanowsky, Guy Bar-Shalom, Fabrizio Frasca, Haggai Maron
The paper introduces Outlier-Aware Attention Linearization (OAL), a method that improves linearized softmax attention by explicitly accounting for a few outlier dimensions that disproportionately affect query-key dot products. OAL builds on a second-order Taylor approximation, uses a Hadamard-product formulation to expose dimension-specific contributions, and groups dimensions with similar scales to share coefficients while treating outliers separately. Experiments on Qwen2.5-0.5B show an 8.9% reduction in frozen-attention output MSE, and fine-tuning with LoRA lowers WikiText-103 perplexity from 31.71 to 27.50 while boosting performance on PIQA, HellaSwag, and ARC-E.
By Weiqi Sun, Qianyue Wu, Wenjia Wang, Tianyang Hu
The paper establishes rigorous trainability results for multi-headed attention layers and Low Rank Adaptation (LoRA) models under stochastic training methods. By proving that the empirical regression loss induces a Poincaré inequality with constants independent of data dimension for LoRA and independent of head dimensions for multi-head attention, the authors show that a stochastic differential equation mimicking SGD converges to the loss minima. These results hold without assumptions on data or model size, providing the first theoretical guarantees for training such architectures.
By Zhengkai Sun, Dibyakanti Kumar, Alejandro F Frangi, Anirbit Mukherjee, Mingfei Sun
The paper introduces the Graph Signature Index to quantify the mismatch between positive-definite geometric priors in many Graph Neural Networks and the negative distances often needed for heterophilic graphs. Using this signature, the authors propose the Pseudo‑Riemannian Attention Network (PRAT), which learns a signed squared pseudo‑distance metric that can capture both attractive and repulsive relationships. Experiments on fifteen node‑classification benchmarks show that PRAT matches or surpasses state‑of‑the‑art heterophily‑adapted models, and ablation studies confirm that the sign structure provided by the signature is essential for its performance.
By Yun Young Choi, Asung Kil, Sun Woo Park, Minho Lee, Seokhwan Kim
arXiv:2605.06462v2 Announce Type: replace
Abstract: Progress in graph learning is hindered by benchmark practices that conflate the contributions of node features and graph structure, making it hard...
By Richard von Moos, Mathieu Alain, Bastian Rieck
arXiv:2604. 17324v2 Announce Type: replace-cross Abstract: Global self-attention drives modern graph transformers, yet the softmax at its core imposes a structural constraint rarely examined directly: every attention row is non-negative and sums to one, so each per-head output is a mass-conserving convex combination of value vectors.
By Yang Liu, Dongxin Guo, Tom Zheng, Siu Ming Yiu, Liam Ning, Jikun Wu
arXiv:2606. 00442v1 Announce Type: new Abstract: Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of modern deep networks.
By Artem Artemev, Rui Xia, Benjamin M. Boyd, Youjing Yu, Felix Dangel, Guillaume Hennequin, Alberto Bernacchia
arXiv:2604. 11613v4 Announce Type: replace-cross Abstract: Transformers can perform in-context classification from a few labeled examples, yet the inference-time algorithm remains opaque.
By Patrick Lutz, Themistoklis Haris, Arjun Chandra, Aditya Gangrade, Venkatesh Saligrama
arXiv:2507. 07814v2 Announce Type: replace Abstract: We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions.
By Nikolay Yudin, Sergei Kudriashov, Alexander Gaponov, Maxim Rakhuba