arXiv Machine Learning
Symmetries and Singularities
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 28
Gating and Learnable Temperature for Graph Attention Mechanisms
The paper introduces gated graph attention and a learnable temperature to enhance standard graph attention networks. Gating filters out unreliable feature dimensions, while the temperature parameter adjusts the sharpness of attention coefficient distributions. Experiments on node classification and heterophilic benchmarks show that these modifications improve performance and robustness, with theoretical analysis supporting the observed gains.
arXiv Machine Learning
Aug 20
Gated Graph Attention Networks with Learnable Temperature
The paper introduces gated graph attention and a learnable temperature to enhance standard graph attention networks. Gated attention filters out unreliable feature dimensions, while the temperature parameter dynamically adjusts the sharpness of attention coefficient distributions. Experiments on both homogeneous and heterophilic heterogeneous benchmarks demonstrate consistent improvements over existing backbones, and controlled noise studies confirm the robustness of these mechanisms.
arXiv Machine Learning
Sep 4
High-Dimensional Learning Dynamics of Attention-Indexed Models
The paper investigates the training dynamics of attention mechanisms in high-dimensional settings, focusing on attention-indexed models that encompass multi-layer and multi-head architectures. It shows that while the loss landscape can be described by a finite set of trace order parameters, the online stochastic gradient descent dynamics involve an infinite hierarchy of matrix moments that can be accurately approximated by a finite truncated system. The study further reveals that the choice of attention parameterization acts as an implicit bias: untied attention can get trapped in uninformative states, whereas tied attention induces symmetry breaking and enables weak recovery with θ(d² log d) samples, and untied attention exhibits a fast-slow dynamic leading to weak recovery when symmetry is broken.
arXiv Machine Learning
1d ago
Graph Metanetworks for Transformer Based Language Models
arXiv:2610.18471v2 Announce Type: replace Abstract: Weight-space learning treats the weights of neural networks as data, processed by other models called metanetworks. A key challenge is that differe...
arXiv AI
Oct 12
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
The paper introduces Outlier-Aware Attention Linearization (OAL), a method that improves linearized softmax attention by explicitly accounting for a few outlier dimensions that disproportionately affect query-key dot products. OAL builds on a second-order Taylor approximation, uses a Hadamard-product formulation to expose dimension-specific contributions, and groups dimensions with similar scales to share coefficients while treating outliers separately. Experiments on Qwen2.5-0.5B show an 8.9% reduction in frozen-attention output MSE, and fine-tuning with LoRA lowers WikiText-103 perplexity from 31.71 to 27.50 while boosting performance on PIQA, HellaSwag, and ARC-E.
arXiv Machine Learning
Sep 10
Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRA
The paper establishes rigorous trainability results for multi-headed attention layers and Low Rank Adaptation (LoRA) models under stochastic training methods. By proving that the empirical regression loss induces a Poincaré inequality with constants independent of data dimension for LoRA and independent of head dimensions for multi-head attention, the authors show that a stochastic differential equation mimicking SGD converges to the loss minima. These results hold without assumptions on data or model size, providing the first theoretical guarantees for training such architectures.