Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

The paper introduces Sliding Networks, a family of residual neural network structures that aim to preserve both feature learning and feature diversity while scaling width and depth. By maintaining a limiting ratio of residual width to the product of hidden width and depth, the authors derive a joint scaling recipe that converges to a stochastic differential equation (SDE) limit with locally maximal feature learning. Experiments on residual-MLP and language models demonstrate that this scaling approach allows redistribution of computational resources between width and depth, though performance varies when parameters are regrouped at a fixed count.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.