← Back to all news
arXiv Machine Learning

Module-Aware Optimization for Graph Neural Networks

The paper introduces GraphMuon, a graph-aware optimizer that extends the matrix-aware Muon optimizer by incorporating the covariance structure of graph‑propagated layer inputs. GraphMuon performs orthogonalization in a metric defined by these propagated representations and normalizes step sizes based on the update’s effect on the layer output, thereby mitigating graph‑induced update amplification. Empirical results show that GraphMuon achieves 1.32×–2.42× faster validation‑loss time‑to‑target compared to AdamW across 15 dataset‑architecture configurations on large‑scale OGB benchmarks, while maintaining competitive predictive performance.

Related stories

arXiv AI
Oct 12

From SGD to Muon: Adaptive Optimization via Schatten-p Norms

The paper introduces a data‑driven criterion that dynamically selects optimal Linear Minimization Oracle (LMO) geometries for each layer of a deep neural network, bridging the gap between SGD and Muon updates. By deriving the criterion from gradient and activation statistics via a single‑step random feature regression surrogate, the authors enable a design space that interpolates between existing optimizers, including SGD, Muon, Adam, and MuAdam. The adaptive approach incurs only about a 3% runtime overhead and demonstrates competitive or superior performance compared to the best static optimizer across three training scenarios.

By Thomas Massena (DTIPG - SNCF, IRIT), Corentin Friedrich (IRIT), Mathieu Serrurier (IRIT)
More like this →
arXiv Machine Learning
Aug 19

Asynchronous Message Passing for Addressing Oversquashing in Graph Neural Networks

The paper introduces an asynchronous message‑passing framework for Graph Neural Networks to mitigate oversquashing, a problem where distant nodes cannot effectively communicate due to structural bottlenecks. Unlike conventional synchronous updates, the method updates a centrality‑guided batch of nodes at each layer, allowing information to propagate sequentially and reducing the need for increased channel capacity. Experiments on six standard and two long‑range graph classification benchmarks show notable performance gains, including 5 % improvement on REDDIT‑BINARY and 4 % on Peptides‑struct.

By Kushal Bose, Swagatam Das
benchmarkssafety
More like this →
arXiv Machine Learning
Oct 3

CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations

CrossGMN introduces a graph metanetwork that processes a trained source network and an initialized target network simultaneously, enabling equivariant cross‑architecture weight‑space transformations. By preserving symmetry through cross‑network message passing, CrossGMN can refine target network initializations while remaining invariant to source permutations and equivariant to target permutations. Experiments demonstrate that CrossGMN accelerates knowledge distillation, transfers across datasets without retraining, and unifies compression from diverse source architectures into a common target architecture.

By Adir Dayan, Yam Eitan, Haggai Maron
llmscomputer-visionefficiency
More like this →