arXiv Machine Learning June 5, 2026 By Hee-Sung Kim, Sungyoon Lee

Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway

Read the original on arXiv Machine Learning →

arXiv:2606. 05219v1 Announce Type: new Abstract: Recent analyses of multi-pathway Deep Linear Networks use Gradient Flow to predict a "winner-takes-all" specialization in which path symmetry breaks and each feature concentrates in a single pathway.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 4

Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks

arXiv:2602. 07494v2 Announce Type: replace Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning.

By Shenxi Wu, Haosong Zhang, Xingjian Ma, Shirui Bian, Yichi Zhang, Xi Chen, Wei Lin

llms

arXiv Machine Learning

Jun 4

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

arXiv:2606. 04754v1 Announce Type: new Abstract: Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged.

By Vincent B\"urgin, Daniel Herbst, Ya-Wei Eileen Lin, Stefanie Jegelka

safety

arXiv Machine Learning

Jun 19

Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

arXiv:2606. 19941v1 Announce Type: new Abstract: Compositionality is believed to be the foundation for generalization, enabling models to reuse meaningful primitives in novel combinations.

By Dat H. Do, Rushi Shah, Duc V. Le, Dianbo Liu

efficiency

arXiv AI

Jul 7

The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

arXiv:2607. 04993v1 Announce Type: cross Abstract: Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them.

By Thomas Hofmann

rag safety

arXiv Machine Learning

Jun 8

Flatland: The Adventures of Gradient Descent with Large Step Sizes

arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.

By Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus, Mark Schmidt, Holger Rauhut

arXiv Machine Learning

Jul 10

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

arXiv:2607. 07884v1 Announce Type: new Abstract: In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth.

By Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara, Andrew Saxe

Related stories

Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

Flatland: The Adventures of Gradient Descent with Large Step Sizes

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks