The paper introduces a new method called uniform‑phase initialization for deep neural networks with sine activations, eliminating the need for the Central Limit Theorem and fully decoupling layers. This approach avoids distributional approximation errors and coupling between layers, leading to stable weight initialization. Experiments show that models using this initialization outperform state‑of‑the‑art methods on image and audio fitting tasks and remain competitive without tuning, while also supporting μP width scaling.
By Simon Kuang, Kyle Chickering, Xinfan Lin
arXiv:2606. 11518v1 Announce Type: cross Abstract: Fourier neural operators (FNOs) are effective and efficient surrogates for approximating solutions of PDEs and generalize across discretizations.
By Pengqing Shi, Jie Yin, Stephen Tierney, Junbin Gao
arXiv:2502. 18959v4 Announce Type: replace Abstract: The architecture of a neural network and the choice of its activation function are both fundamental to its performance.
By Shijun Zhang, Hongkai Zhao, Yimin Zhong, Haomin Zhou
The paper investigates how two‑layer polynomial‑width neural networks learn orthogonal multi‑index targets under standard initialization. It shows that incremental learning still occurs: the loss decreases sequentially following the Hermite expansion, with lower‑order components learned first. The dynamics also exhibit a competitive reallocation of parameter mass, shifting into the target subspace and concentrating on aligned neurons. The analysis uses a symmetry‑based finite‑width approximation and demonstrates that vanilla gradient descent displays the same qualitative behavior.
By Mo Zhou, Weihang Xu, Simon S. Du, Maryam Fazel
arXiv:2606. 23129v2 Announce Type: replace-cross Abstract: Implicit Neural Representations (INRs) have been proven successful in encoding continuous signals through coordinate-based networks, yet facing a spectral dilemma: periodic activations capture fine details but act as all-pass filters that memorise noise, while spatially compact activations regularise effectively but suffer from low-frequency bias.
By Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti, Luigi Di Stefano
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
By Berk Tinaz, Changzhi Xie, Mahdi Soltanolkotabi
arXiv:2608. 14733v1 Announce Type: cross Abstract: Building on the foundation of single-hidden-layer neural networks, Fourier Feature Networks (FENs) are proposed, which incorporate Fourier features using $\cos$, $\sin$, or a combination of both.
By Qihong Yang, Zhijie Su, Yangtao Deng, Qiaolin He
arXiv:2607. 12332v1 Announce Type: new Abstract: We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization.
By Jiajie Zhao, Jianxing Wang, Junjie Yang, Zhiwei Bai, Yaoyu Zhang
arXiv:2606. 16575v1 Announce Type: new Abstract: Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors.
By Yong Wang, Tao Zhou, Xuhui Meng
We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization. Extending Theorem 1 from Pesme & Flammarion (2023), we generalize the analysis to both deep diagonal linear networks and a broader class of two-layer diagonal linear networks (as defined in Definition 4.
arXiv:2608. 11970v1 Announce Type: new Abstract: The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions.
By Daehwa Ko, Jaehyeon Kim, Seunghyun Ham, Jay Hoon Jung
arXiv:2606. 16575v2 Announce Type: replace Abstract: Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors.
By Yong Wang, Tao Zhou, Xuhui Meng