arXiv AI
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 16
Scaling Adaptive Depth with Norm-Agnostic Residual Networks
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
arXiv Machine Learning
Sep 1
Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets
arXiv:2403.04545v4 Announce Type: replace Abstract: Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free a...
arXiv AI
Jul 1
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
arXiv:2606. 30813v1 Announce Type: cross Abstract: Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training.
arXiv Machine Learning
Aug 4
Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks
arXiv:2602. 07494v2 Announce Type: replace Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning.
arXiv Machine Learning
Jul 10
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
arXiv:2607. 07884v1 Announce Type: new Abstract: In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth.
arXiv Machine Learning
Aug 18
Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion
arXiv:2608. 14664v1 Announce Type: new Abstract: How can we determine whether a trained neural network is already deep enough?